Reading SAS and clinical datasets

Lesson 8 — Python

Lesson 8 of 20 Intermediate ~75 min

Learning objectives

  • Read .sas7bdat and .xpt files in Python
  • Preserve and use SAS metadata — labels, formats, value labels
  • Write XPT files that meet transport constraints
  • Handle encoding and large files
  • Assess what the Python clinical ecosystem can and cannot do
  • Interoperate with R

The options

Library Reads Writes Metadata
pandas.read_sas sas7bdat, xpt No Minimal
pyreadstat sas7bdat, xpt, sav, dta xpt, sav, dta Full
xport xpt xpt Partial
sas7bdat sas7bdat No Some

pyreadstat is the right default. It is built on the same ReadStat C library as R’s haven, so the two produce consistent results — which matters when an R programmer and a Python programmer QC each other’s work.

pip install pyreadstat

Reading

import pyreadstat

df, meta = pyreadstat.read_sas7bdat("data/raw/sdtm/dm.sas7bdat")
df, meta = pyreadstat.read_xport("data/raw/sdtm/ae.xpt")

The second return value carries everything pandas.read_sas throws away:

meta.column_names
#> ['STUDYID', 'USUBJID', 'SUBJID', 'AGE', 'SEX', ...]

meta.column_labels
#> ['Study Identifier', 'Unique Subject Identifier', ..., 'Age', 'Sex']

meta.column_names_to_labels
#> {'STUDYID': 'Study Identifier', 'AGE': 'Age', ...}

meta.variable_value_labels
#> {'RACEN': {1.0: 'WHITE', 2.0: 'BLACK', 3.0: 'ASIAN'}}

meta.original_variable_types
#> {'AGE': 'BEST', 'USUBJID': '$14'}

meta.variable_measure          # nominal / ordinal / scale
meta.file_label                # dataset label
meta.number_rows, meta.number_columns
meta.file_encoding

A metadata table

import pandas as pd


def sas_metadata(df: pd.DataFrame, meta) -> pd.DataFrame:
    """Build a data dictionary from a pyreadstat read."""
    return pd.DataFrame({
        "variable":   meta.column_names,
        "label":      meta.column_labels,
        "sas_format": [meta.original_variable_types.get(c)
                       for c in meta.column_names],
        "dtype":      [str(df[c].dtype) for c in meta.column_names],
        "n_missing":  [int(df[c].isna().sum()) for c in meta.column_names],
        "n_distinct": [int(df[c].nunique()) for c in meta.column_names],
        "max_bytes":  [
            int(df[c].dropna().astype(str).str.encode("utf-8").str.len().max())
            if df[c].dtype == object and df[c].notna().any() else None
            for c in meta.column_names
        ],
    })

max_bytes measured in bytes, not characters — the XPT limit is a byte limit and a non-ASCII character occupies several.

Applying value labels

# Read with value labels applied as pandas categoricals
df, meta = pyreadstat.read_sas7bdat("dm.sas7bdat", apply_value_formats=True)

# Or apply afterwards, selectively
df["RACE"] = df["RACEN"].map(meta.variable_value_labels["RACEN"])

# Or all at once
for col, mapping in meta.variable_value_labels.items():
    if col in df.columns:
        df[col + "_LBL"] = df[col].map(mapping)
WarningLabels do not survive pandas operations 
df.attrs["column_labels"] = meta.column_names_to_labels
df2 = df.assign(AGE=df["AGE"] + 1)
df2.attrs
#> {}       — attrs is not propagated reliably

Like R, pandas drops metadata through most operations — and unlike R, there is no xportr to reapply it from a specification. Carry the metadata separately in a dict and reapply at write time. This is the main practical gap versus the R ecosystem.

Performance

# Only the columns you need
df, meta = pyreadstat.read_sas7bdat(
    "lb.sas7bdat",
    usecols=["USUBJID", "LBTESTCD", "LBSTRESN", "LBDTC"],
)

# Metadata only — instant, even on a 2 GB file
_, meta = pyreadstat.read_sas7bdat("lb.sas7bdat", metadataonly=True)
print(f"{meta.number_rows:,} rows, {meta.number_columns} columns")

# Multiprocessing
df, meta = pyreadstat.read_file_multiprocessing(
    pyreadstat.read_sas7bdat, "lb.sas7bdat", num_processes=4,
)

# Chunked, for files that do not fit in memory
reader = pyreadstat.read_file_in_chunks(
    pyreadstat.read_sas7bdat, "lb.sas7bdat", chunksize=100_000,
)
for chunk, meta in reader:
    process(chunk)

metadataonly=True is the fastest way to inventory a directory of SAS files.

Encoding

df, meta = pyreadstat.read_sas7bdat("ae.sas7bdat", encoding="latin1")
df, meta = pyreadstat.read_sas7bdat("ae.sas7bdat", encoding="WINDOWS-1252")

meta.file_encoding
#> 'WINDOWS-1252'

Detect problems:

def find_non_ascii(df: pd.DataFrame) -> pd.DataFrame:
    """Report columns containing non-ASCII characters."""
    rows = []
    for col in df.select_dtypes(include=["object", "string"]).columns:
        s = df[col].dropna().astype(str)
        mask = s.str.contains(r"[^\x00-\x7F]", regex=True, na=False)
        if mask.any():
            rows.append({
                "variable": col,
                "n": int(mask.sum()),
                "examples": " | ".join(s[mask].unique()[:3]),
            })
    return pd.DataFrame(rows)

Caching to a fast format

Reading SAS files is slow. Convert once:

from pathlib import Path
import json

def cache_sdtm(sdtm_dir: str, cache_dir: str) -> None:
    """Convert SAS files to Parquet, keeping the metadata as JSON."""
    src, dst = Path(sdtm_dir), Path(cache_dir)
    dst.mkdir(parents=True, exist_ok=True)

    for f in sorted(src.glob("*.sas7bdat")):
        df, meta = pyreadstat.read_sas7bdat(str(f))

        df.to_parquet(dst / f"{f.stem}.parquet", index=False)

        (dst / f"{f.stem}_meta.json").write_text(json.dumps({
            "labels":       meta.column_names_to_labels,
            "value_labels": {k: {str(kk): vv for kk, vv in v.items()}
                             for k, v in meta.variable_value_labels.items()},
            "file_label":   meta.file_label,
            "n_rows":       meta.number_rows,
        }, indent=2))

        print(f"{f.name}: {meta.number_rows:,} rows")

Parquet reads roughly an order of magnitude faster and preserves dtypes exactly. The metadata JSON preserves what Parquet cannot.

Writing XPT

import pyreadstat

pyreadstat.write_xport(
    df,
    "data/submission/adsl.xpt",
    file_format_version=5,
    table_name="ADSL",
    file_label="Subject-Level Analysis Dataset",
    column_labels=[spec_labels[c] for c in df.columns],
    variable_format={"TRTSDT": "DATE9.", "TRTEDT": "DATE9."},
)

The V5 constraints from the R lesson apply identically, and Python has no xportr. Write the checks yourself:

def check_xpt_v5(df: pd.DataFrame, labels: dict[str, str]) -> list[str]:
    """Check a DataFrame against SAS transport V5 constraints."""
    problems = []

    # Variable names: 8 characters, and no collisions on truncation
    long_names = [c for c in df.columns if len(c) > 8]
    if long_names:
        problems.append(f"Names over 8 characters: {long_names}")

    stems = pd.Series([c[:8].upper() for c in df.columns])
    dupes = stems[stems.duplicated()].unique().tolist()
    if dupes:
        colliding = [c for c in df.columns if c[:8].upper() in dupes]
        problems.append(f"Names collide when truncated: {colliding}")

    # Labels: 40 characters
    long_labels = {c: lbl for c, lbl in labels.items() if len(lbl) > 40}
    if long_labels:
        problems.append(f"Labels over 40 characters: {list(long_labels)}")

    # Character values: 200 bytes
    for col in df.select_dtypes(include=["object", "string"]).columns:
        s = df[col].dropna().astype(str)
        if s.empty:
            continue
        max_bytes = s.str.encode("utf-8").str.len().max()
        if max_bytes > 200:
            problems.append(f"{col}: values up to {max_bytes} bytes (max 200)")

    # ASCII only
    non_ascii = find_non_ascii(df)
    if len(non_ascii):
        problems.append(f"Non-ASCII in: {non_ascii['variable'].tolist()}")

    return problems


problems = check_xpt_v5(adsl, labels)
if problems:
    raise ValueError("XPT V5 violations:\n  " + "\n  ".join(problems))
ImportantName truncation collides silently

TREATMENT_START and TREATMENT_STOP both truncate to TREATMEN. Whichever is written second overwrites the first, and nothing warns you. The collision check above is not optional.

The Python clinical ecosystem

An honest assessment as of 2026.

Capability R Python
Read SAS files haven — mature pyreadstat — mature
Write XPT haven + xportr pyreadstat — no conformance layer
ADaM derivations admiral — comprehensive Nothing equivalent
Metadata / specifications metacore, metatools Nothing equivalent
Transport conformance xportr Hand-rolled
Table computation Tplyr, rtables Polars/pandas by hand
RTF output r2rtf rtflite — equivalent
DOCX assembly officer rtflite[docx]
define.xml defineR, commercial Nothing equivalent
Statistical modelling Comprehensive statsmodels, lifelines — good
Machine learning Adequate scikit-learn — much stronger
Data engineering Adequate Much stronger
Interactive apps Shiny — mature Streamlit, Shiny — good

The gap is in CDISC-specific tooling, and it is narrowing unevenly.

Presentation is solved. rtflite is a pharmaverse package that produces submission-quality RTF from Python, written by the author of r2rtf and mirroring its design. If your ADaM datasets already exist, a Python TLF pipeline is now a reasonable choice — see Clinical tables and TLF generation.

Derivation is not. There is still no admiral, no metacore, no xportr. Building ADaM in Python means writing from scratch what admiral gives you tested, as lesson 9 demonstrates.

That split shapes the sensible division of labour:

Python is the better tool for: data engineering and ETL, machine learning, API integration, image and signal analysis, wearable and sensor data, large-scale text processing, and anything that has to integrate with a production software system.

R is the better tool for: SDTM and ADaM derivation, metadata-driven programming, transport conformance, and define.xml — everywhere the pharmaverse already solved the problem.

Either works for: TLF production, now that rtflite exists.

Many organisations use both, and that is a defensible architecture rather than a failure to decide.

Interoperating with R

Files

The simplest and most robust approach:

# Python writes
df.to_parquet("data/shared/adsl.parquet", index=False)
# R reads
adsl <- arrow::read_parquet("data/shared/adsl.parquet")

Parquet preserves types exactly, is fast, and neither side needs the other installed. This is the default recommendation.

Feather / Arrow IPC

df.to_feather("data/shared/adsl.feather")
adsl <- arrow::read_feather("data/shared/adsl.feather")

Slightly faster than Parquet for short-lived interchange, less compressed.

reticulate — calling Python from R

library(reticulate)
use_virtualenv("~/.venvs/study")

pd <- import("pandas")
sklearn <- import("sklearn.ensemble")

adsl_py <- r_to_py(adsl)
model <- sklearn$RandomForestClassifier(n_estimators = 500L)
model$fit(X, y)
predictions <- py_to_r(model$predict(X_new))

Or a Python chunk in a Quarto document:

```{python}
import pandas as pd
result = pd.DataFrame({"x": [1, 2, 3]})
```

```{r}
py$result
```

rpy2 — calling R from Python

import rpy2.robjects as ro
from rpy2.robjects import pandas2ri
from rpy2.robjects.packages import importr

pandas2ri.activate()

admiral = importr("admiral")
adsl_r = ro.conversion.py2rpy(adsl_df)
result = admiral.derive_var_trtdurd(adsl_r)
adsl_out = ro.conversion.rpy2py(result)

This lets a Python pipeline call admiral. It works, but it adds a substantial dependency and a debugging surface across two runtimes. Consider whether a file handoff would do instead.

R and Python side by side

Task R Python
Read sas7bdat haven::read_sas() pyreadstat.read_sas7bdat()
Read XPT haven::read_xpt() pyreadstat.read_xport()
Write XPT haven::write_xpt() pyreadstat.write_xport()
Get labels attr(x, "label") meta.column_labels
Value labels attr(x, "labels") meta.variable_value_labels
Apply formats haven::as_factor() apply_value_formats=True
Strip metadata zap_labels() Already absent
Transport conformance xportr Hand-rolled
Fast cache saveRDS, arrow to_parquet

Common mistakes

Mistake Consequence Fix
pandas.read_sas for clinical work Metadata lost pyreadstat
Assuming labels survive operations Silently dropped Carry them separately
Measuring width in characters XPT violations .str.encode("utf-8").str.len()
No V5 conformance checks Non-conformant files Write the checks
Ignoring name truncation collisions One variable overwrites another Explicit collision check
Re-reading SAS files repeatedly Very slow pipelines Cache to Parquet
Building an ADaM pipeline from scratch Reinventing admiral, untested Consider R for this layer

Exercise 7.1 — SDTM inventory

Write a function that scans a directory of SAS files and returns a DataFrame with one row per domain: domain, file size, rows, columns, whether USUBJID is present, distinct subjects, and whether --SEQ is unique. Use metadataonly where possible.

Show solution 
from pathlib import Path
import pandas as pd
import pyreadstat


def sdtm_inventory(sdtm_dir: str, full_read: bool = True) -> pd.DataFrame:
    """Inventory a directory of SDTM datasets.

    Args:
        sdtm_dir: Directory containing .sas7bdat or .xpt files.
        full_read: If False, only metadata is read — much faster, but
            subject counts and key uniqueness are unavailable.
    """
    path = Path(sdtm_dir)
    files = sorted([*path.glob("*.sas7bdat"), *path.glob("*.xpt")])

    if not files:
        raise FileNotFoundError(f"No SAS files found in {path}")

    rows = []
    for f in files:
        domain = f.stem.upper()
        reader = (pyreadstat.read_xport if f.suffix == ".xpt"
                  else pyreadstat.read_sas7bdat)

        record = {
            "domain":   domain,
            "file":     f.name,
            "size_mb":  round(f.stat().st_size / 1024**2, 2),
        }

        try:
            # Metadata is cheap even on a 2 GB file
            _, meta = reader(str(f), metadataonly=True)
            record.update({
                "n_rows":      meta.number_rows,
                "n_variables": meta.number_columns,
                "file_label":  meta.file_label,
                "encoding":    meta.file_encoding,
                "has_usubjid": "USUBJID" in meta.column_names,
            })

            seq_var = f"{domain}SEQ"
            record["seq_var"] = seq_var if seq_var in meta.column_names else None

            if full_read and record["has_usubjid"]:
                cols = ["USUBJID"] + ([record["seq_var"]]
                                      if record["seq_var"] else [])
                df, _ = reader(str(f), usecols=cols)

                record["n_subjects"] = int(df["USUBJID"].nunique())
                record["seq_unique"] = (
                    not df.duplicated(subset=cols).any()
                    if record["seq_var"] else None
                )
            else:
                record["n_subjects"] = None
                record["seq_unique"] = None

            record["status"] = "OK"

        except Exception as e:
            record.update({
                "n_rows": None, "n_variables": None, "n_subjects": None,
                "seq_var": None, "seq_unique": None,
                "status": f"FAILED: {type(e).__name__}: {e}",
            })

        rows.append(record)

    return pd.DataFrame(rows).sort_values("domain").reset_index(drop=True)
inv = sdtm_inventory("data/raw/sdtm")
print(inv.to_string(index=False))
 domain          file  size_mb  n_rows  n_variables               file_label encoding  has_usubjid seq_var  n_subjects  seq_unique status
     AE   ae.sas7bdat     2.41    1847           31          Adverse Events    UTF-8         True   AESEQ         218        True     OK
     DM   dm.sas7bdat     0.18     306           24            Demographics    UTF-8         True    None         306        None     OK
     EX   ex.sas7bdat     0.94     892           26                Exposure    UTF-8         True   EXSEQ         254        True     OK
     LB   lb.sas7bdat    48.20  124903           38     Laboratory Test Res    UTF-8         True   LBSEQ         298       False     OK

Two things the inventory surfaces immediately:

  • LB has seq_unique = False. LBSEQ is not unique within USUBJID, which is a conformance violation and will cause row multiplication in any join. Raise it with data management before writing a line of derivation code.

  • DM correctly has no --SEQ. It is one record per subject, so USUBJID alone is the key, and n_subjects == n_rows confirms it.

The two-tier read is what makes this usable on a real study: metadataonly=True is instant even on the 48 MB LB file, and the second read pulls only the two columns needed for the key check rather than all 38.

# Fast path for a first look at an unfamiliar directory
sdtm_inventory("data/raw/sdtm", full_read=False)
The try/except around each file matters too — one corrupt or unreadable file should not prevent inventorying the other eleven, and recording why it failed is more useful than a traceback.

Exercise 7.2 — XPT round trip with conformance checks

Write a function that takes a DataFrame and a specification, runs every V5 conformance check, writes the XPT, reads it back and reports any differences.

Show solution 
from pathlib import Path
import pandas as pd
import pyreadstat


def write_xpt_checked(
    df: pd.DataFrame,
    spec: pd.DataFrame,
    path: str,
    table_name: str,
    file_label: str = "",
    strict: bool = True,
) -> pd.DataFrame:
    """Write an XPT V5 file with conformance checks and a round-trip comparison.

    Args:
        df: The dataset to write.
        spec: Specification with columns variable, label, length, format.
        path: Output path.
        table_name: SAS dataset name, 8 characters or fewer.
        file_label: Dataset label, 40 characters or fewer.
        strict: Raise on any violation. If False, warn instead.

    Returns:
        A DataFrame of differences found in the round trip (empty if none).
    """
    problems: list[str] = []

    # ---- 1. Table name and file label -----------------------------------
    if len(table_name) > 8:
        problems.append(f"table_name '{table_name}' exceeds 8 characters")
    if len(file_label) > 40:
        problems.append(f"file_label exceeds 40 characters ({len(file_label)})")

    # ---- 2. Variable names ------------------------------------------------
    long_names = [c for c in df.columns if len(c) > 8]
    if long_names:
        problems.append(f"Names over 8 characters: {long_names}")

    stems = pd.Series([c[:8].upper() for c in df.columns])
    collisions = stems[stems.duplicated()].unique().tolist()
    if collisions:
        affected = [c for c in df.columns if c[:8].upper() in collisions]
        problems.append(f"Names collide when truncated to 8: {affected}")

    invalid = [c for c in df.columns
               if not c[0].isalpha() and not c.startswith("_")]
    if invalid:
        problems.append(f"Names must start with a letter or underscore: {invalid}")

    # ---- 3. Specification coverage ---------------------------------------
    spec_vars = set(spec["variable"])
    data_vars = set(df.columns)
    if data_vars - spec_vars:
        problems.append(f"In data but not spec: {sorted(data_vars - spec_vars)}")
    if spec_vars - data_vars:
        problems.append(f"In spec but not data: {sorted(spec_vars - data_vars)}")

    # ---- 4. Labels ---------------------------------------------------------
    labels = dict(zip(spec["variable"], spec["label"]))
    long_labels = {v: l for v, l in labels.items() if isinstance(l, str) and len(l) > 40}
    if long_labels:
        problems.append(f"Labels over 40 characters: {list(long_labels)}")

    # ---- 5. Character widths and ASCII ------------------------------------
    for col in df.select_dtypes(include=["object", "string"]).columns:
        s = df[col].dropna().astype(str)
        if s.empty:
            continue

        max_bytes = int(s.str.encode("utf-8").str.len().max())
        if max_bytes > 200:
            worst = s.loc[s.str.encode("utf-8").str.len().idxmax()]
            problems.append(
                f"{col}: max {max_bytes} bytes (limit 200) — '{worst[:60]}...'"
            )

        spec_len = spec.loc[spec["variable"] == col, "length"]
        if len(spec_len) and pd.notna(spec_len.iloc[0]) and max_bytes > spec_len.iloc[0]:
            problems.append(
                f"{col}: {max_bytes} bytes exceeds specified length "
                f"{int(spec_len.iloc[0])} — values WILL be truncated"
            )

        if s.str.contains(r"[^\x00-\x7F]", regex=True, na=False).any():
            examples = s[s.str.contains(r"[^\x00-\x7F]", regex=True)].unique()[:2]
            problems.append(f"{col}: non-ASCII characters, e.g. {list(examples)}")

    # ---- Report ------------------------------------------------------------
    if problems:
        message = "XPT V5 conformance issues:\n  " + "\n  ".join(problems)
        if strict:
            raise ValueError(message)
        import warnings
        warnings.warn(message)

    # ---- Write --------------------------------------------------------------
    Path(path).parent.mkdir(parents=True, exist_ok=True)

    pyreadstat.write_xport(
        df, path,
        file_format_version=5,
        table_name=table_name,
        file_label=file_label,
        column_labels=[labels.get(c, "") for c in df.columns],
    )

    # ---- Round trip ---------------------------------------------------------
    returned, meta = pyreadstat.read_xport(path)

    diffs = []

    if len(returned) != len(df):
        diffs.append({"issue": "row_count", "variable": None,
                      "before": len(df), "after": len(returned)})

    for col in df.columns:
        if col not in returned.columns:
            diffs.append({"issue": "column_missing", "variable": col,
                          "before": "present", "after": "absent"})
            continue

        if df[col].dtype == object:
            before = df[col].dropna().astype(str)
            after = returned[col].dropna().astype(str).str.rstrip()  # SAS pads
            if not before.reset_index(drop=True).equals(after.reset_index(drop=True)):
                n_diff = (before.reset_index(drop=True)
                          != after.reset_index(drop=True)).sum()
                first = before[before.reset_index(drop=True)
                               != after.reset_index(drop=True)].iloc[0]
                diffs.append({"issue": "value_changed", "variable": col,
                              "before": first[:50],
                              "after": f"{n_diff} value(s) differ"})
        else:
            if not df[col].astype(float).equals(returned[col].astype(float)):
                diffs.append({"issue": "numeric_changed", "variable": col,
                              "before": None, "after": None})

    result = pd.DataFrame(diffs)

    if result.empty:
        print(f"✔ {path}: written and verified ({len(df):,} rows, "
              f"{len(df.columns)} variables)")
    else:
        print(f"✘ {path}: {len(result)} difference(s) after round trip")
        print(result.to_string(index=False))

    return result

Used:

spec = pd.DataFrame({
    "variable": ["USUBJID", "AGE", "SEX", "TRT01P"],
    "label":    ["Unique Subject Identifier", "Age", "Sex",
                 "Planned Treatment for Period 01"],
    "length":   [30, 8, 1, 30],
    "format":   [None, None, None, None],
})

diffs = write_xpt_checked(
    adsl, spec,
    path="data/submission/adsl.xpt",
    table_name="ADSL",
    file_label="Subject-Level Analysis Dataset",
)

The critical part is checking max_bytes > spec_len before writing. XPT truncation is silent — pyreadstat writes the file, no error is raised, and the data loss is only discoverable by reading the file back. That check turns an invisible data-integrity failure into an immediate, located error naming the offending column and value.

The round-trip comparison catches what the pre-flight checks miss: trailing blank padding (handled with .str.rstrip()), type changes, and anything pyreadstat does that is not documented. Running it once per study on the first dataset is enough to build confidence; running it on every dataset costs seconds and is worth it.

This function is doing by hand what R’s xportr provides as a tested package — which is a fair illustration of the ecosystem gap described above.

Recap

  • pyreadstat, not pandas.read_sas — it preserves labels, formats and value labels
  • Metadata does not survive pandas operations; carry it separately in a dict
  • metadataonly=True and usecols= for fast inventory and selective reads
  • Cache to Parquet plus a metadata JSON
  • Measure character widths in bytes; check 8-character name collisions explicitly
  • Python has no admiral, xportr, Tplyr or r2rtf — that gap is real
  • Parquet file handoff is the simplest and most robust R/Python interop

Next: Building CDISC datasets in Python.

Back to top