Validation and testing
Lesson 8 — Clinical Programming with R
Learning objectives
- Distinguish validation, verification and QC
- Run an independent double-programming comparison with
diffdf - Write unit tests for derivations and trace them to requirements
- Assess whether an R package is fit for regulated use
- Produce the evidence a QC or audit review expects
Three words that get confused
| Term | Question | Who |
|---|---|---|
| Validation | Does the system do what it is intended to do, in its intended environment? | Quality function, system level |
| Verification | Does the output match the specification? | QC programmer, output level |
| Testing | Does this code do what the programmer intended? | Programmer, code level |
Unit tests are testing. Independent double programming is verification. Neither, on its own, is validation — validation is a documented, system-level activity that includes both plus environment qualification, change control and review records.
Independent double programming
The industry’s primary verification method: two programmers implement the same specification independently and compare outputs.
Specification (SAP + ADaM spec)
│
├──▶ Production programmer ──▶ adsl.rds
│
└──▶ QC programmer ──────────▶ adsl_qc.rds
│
compare ──▶ differences ──▶ resolve“Independent” means the QC programmer works from the specification, not from the production code. Reading the production code first — even briefly — defeats the purpose, because misreadings of the specification propagate.
diffdf
library(diffdf)
prod <- readRDS("data/adam/adsl.rds")
qc <- readRDS("data/adam_qc/adsl.rds")
diffdf(prod, qc, keys = "USUBJID")Differences found between the objects!
Not all Values Compared Equal
Variable No of Differences
TRTDURD 3
AGEGR1 12
All rows are shown in table below
==========================================
VARIABLE USUBJID BASE COMPARE
------------------------------------------
TRTDURD 01-701-1015 182 183
TRTDURD 01-701-1023 170 171
AGEGR1 01-703-1096 65-80 >80
==========================================Options:
diffdf(
prod, qc,
keys = c("USUBJID", "PARAMCD", "AVISITN"),
suppress_warnings = FALSE,
tolerance = 1e-8, # numeric comparison tolerance
strict_numeric = FALSE, # allow integer vs double
strict_factor = FALSE, # allow factor vs character
file = "output/qc/diffdf_adsl.txt"
)
# Programmatic use
result <- diffdf(prod, qc, keys = "USUBJID")
diffdf_has_issues(result)
#> [1] TRUEtolerance is not a way to make differences disappear
Setting tolerance = 0.01 to silence a difference of 0.008 hides a real discrepancy. Numeric differences at the 1e-8 level are floating-point noise; differences at the 1e-3 level are two different calculations.
Investigate before adjusting the tolerance. The usual causes of genuine floating-point noise are different operation orders and different rounding points — both worth understanding rather than suppressing.
Interpreting differences
| Difference | Usual cause |
|---|---|
| Different row counts | One programmer filtered differently, or a join fanned out |
| A variable in one and not the other | Spec ambiguity about whether it is required |
| Systematic off-by-one in a day variable | Day zero convention |
| Differences only for partial dates | Imputation convention |
| Differences in the last decimal only | Rounding convention (half-even vs half-up) |
| Differences for a handful of subjects | Genuine edge case — this is the valuable one |
Every difference must be resolved and documented, with a decision: production was wrong, QC was wrong, or the specification was ambiguous. The third outcome is common and is the most valuable result of the exercise — an ambiguous spec would have caused a problem eventually.
QC comparison at scale
compare_all <- function(prod_dir, qc_dir, keys_map) {
datasets <- intersect(
tools::file_path_sans_ext(list.files(prod_dir, "\\.rds$")),
tools::file_path_sans_ext(list.files(qc_dir, "\\.rds$"))
)
purrr::map(datasets, function(ds) {
p <- readRDS(file.path(prod_dir, paste0(ds, ".rds")))
q <- readRDS(file.path(qc_dir, paste0(ds, ".rds")))
res <- diffdf(p, q, keys = keys_map[[ds]],
file = file.path("output/qc", paste0("diffdf_", ds, ".txt")),
suppress_warnings = TRUE)
tibble::tibble(
dataset = ds,
prod_rows = nrow(p),
qc_rows = nrow(q),
prod_cols = ncol(p),
qc_cols = ncol(q),
has_issues = diffdf_has_issues(res),
status = if (diffdf_has_issues(res)) "DIFFERENCES" else "MATCH"
)
}) |>
purrr::list_rbind()
}
compare_all("data/adam", "data/adam_qc",
keys_map = list(
adsl = "USUBJID",
adae = c("USUBJID", "AESEQ"),
adlb = c("USUBJID", "PARAMCD", "AVISITN", "ADT")
))
#> # A tibble: 3 x 7
#> dataset prod_rows qc_rows prod_cols qc_cols has_issues status
#> <chr> <int> <int> <int> <int> <lgl> <chr>
#> 1 adsl 306 306 58 58 FALSE MATCH
#> 2 adae 1191 1191 45 45 TRUE DIFFERENCES
#> 3 adlb 20983 20983 62 62 FALSE MATCHComparing tables
For TLFs, compare the computed numbers, not the RTF:
prod_tbl <- readRDS("output/tables/t_14_1_1_data.rds")
qc_tbl <- readRDS("output/qc/t_14_1_1_data.rds")
diffdf(prod_tbl, qc_tbl, keys = "label")This is why lesson 6 recommends saving the numeric result alongside every table. Comparing RTF files byte by byte fails on timestamps and tells you nothing about which number differs.
Unit testing derivations
Double programming verifies one dataset on one day. Unit tests verify the logic, every time anything changes, in seconds.
# tests/testthat/test-derive-trtemfl.R
test_that("REQ-042: events on or after first dose are treatment emergent", {
ae <- make_ae(astdt = c("2026-03-15", "2026-04-01"))
out <- derive_trtemfl(ae, trtsdt = as.Date("2026-03-15"),
trtedt = as.Date("2026-06-15"))
expect_equal(out$TRTEMFL, c("Y", "Y"))
})
test_that("REQ-042: events before first dose are not treatment emergent", {
ae <- make_ae(astdt = c("2026-03-14", "2026-01-01"))
out <- derive_trtemfl(ae, trtsdt = as.Date("2026-03-15"),
trtedt = as.Date("2026-06-15"))
expect_equal(out$TRTEMFL, c("N", "N"))
})
test_that("REQ-043: the 30-day follow-up window is inclusive", {
ae <- make_ae(astdt = c("2026-07-15", "2026-07-16")) # TRTEDT + 30, +31
out <- derive_trtemfl(ae, trtsdt = as.Date("2026-03-15"),
trtedt = as.Date("2026-06-15"))
expect_equal(out$TRTEMFL, c("Y", "N"))
})
test_that("REQ-044: subjects with no treatment start are flagged N", {
ae <- make_ae(astdt = "2026-04-01")
out <- derive_trtemfl(ae, trtsdt = as.Date(NA), trtedt = as.Date(NA))
expect_equal(out$TRTEMFL, "N")
})Putting the requirement ID in the test name makes the traceability matrix a grep:
traceability <- function(test_dir = "tests/testthat") {
files <- list.files(test_dir, pattern = "^test-", full.names = TRUE)
purrr::map(files, function(f) {
lines <- readLines(f)
tests <- grep('^\\s*test_that\\(', lines, value = TRUE)
tibble::tibble(
file = basename(f),
test = stringr::str_match(tests, 'test_that\\("(.*?)"')[, 2]
)
}) |>
purrr::list_rbind() |>
mutate(requirement = stringr::str_extract(test, "REQ-\\d+")) |>
filter(!is.na(requirement)) |>
arrange(requirement)
}
traceability()
#> # A tibble: 24 x 3
#> file test requirement
#> <chr> <chr> <chr>
#> 1 test-derive-trtemfl.R REQ-042: events on or after first dose are... REQ-042
#> 2 test-derive-trtemfl.R REQ-042: events before first dose are not... REQ-042Then check that every requirement has at least one test:
requirements <- readr::read_csv("docs/requirements.csv")
untested <- setdiff(requirements$id, traceability()$requirement)
if (length(untested) > 0) {
cli::cli_warn("Untested requirement{?s}: {.val {untested}}")
}Assessing packages for regulated use
Before adding a package to a study, some due diligence.
library(riskmetric)
pkg_assess(pkg_ref("admiral")) |> pkg_score()
#> # A tibble: 1 x 3
#> package version pkg_score
#> <chr> <chr> <dbl>
#> 1 admiral 1.1.1 0.087 (lower is lower risk)riskmetric assesses: test coverage, documentation completeness, bug closure rate, download counts, maintainer responsiveness, source control presence, R CMD check results, licence.
It is a screening tool, not a validation. What a risk assessment should actually consider:
| Dimension | Questions |
|---|---|
| Provenance | Who maintains it? CRAN or a private repo? |
| Testing | Coverage? Are the tests meaningful? Do they run in CI? |
| Documentation | Complete help pages? Vignettes? A specification? |
| Community | Downloads, open issues, time to close bugs |
| Stability | How often does the interface break? Deprecation policy? |
| Purpose | Is it used for a derivation that affects results, or for plotting? |
| Alternatives | Could you write the 20 lines yourself and test them? |
A pragmatic tiering:
| Tier | Examples | Assessment |
|---|---|---|
| Base and recommended | stats, utils, survival |
Accepted; part of R |
| Core, high-scrutiny | dplyr, tidyr, haven, admiral, xportr |
Full risk assessment, version pinned |
| Supporting | ggplot2, stringr, lubridate |
Lighter assessment |
| Convenience | Anything used only interactively | Not in the production environment |
The tier depends on what the package does, not what it is. ggplot2 used to make an exploratory plot is low risk; ggplot2 used to produce a figure in the CSR is not.
Pin every version:
renv::snapshot()And record the assessment outcome alongside the study documentation.
QC evidence
What a reviewer or auditor expects to see:
| Evidence | Artefact |
|---|---|
| Independent programming performed | QC program, in version control, by a different author |
| Comparison performed | diffdf output file, dated |
| Differences resolved | A log with each difference, its cause and its resolution |
| Requirements tested | Traceability matrix, test results |
| Environment recorded | sessionInfo(), renv.lock, container digest |
| Code reviewed | Pull request with a recorded approval |
| Change controlled | Every change linked to a request |
Generate what you can:
qc_report <- function(dataset, prod, qc, keys, output_dir = "output/qc") {
result <- diffdf(prod, qc, keys = keys, suppress_warnings = TRUE)
report <- list(
dataset = dataset,
date = Sys.time(),
performed_by = Sys.info()[["user"]],
prod_file = attr(prod, "provenance")$git_sha %||% NA,
qc_file = attr(qc, "provenance")$git_sha %||% NA,
prod_rows = nrow(prod), qc_rows = nrow(qc),
prod_vars = ncol(prod), qc_vars = ncol(qc),
result = if (diffdf_has_issues(result)) "DIFFERENCES FOUND" else "MATCH",
r_version = R.version.string,
key_packages = purrr::map_chr(
c("admiral", "dplyr", "haven"),
~ as.character(utils::packageVersion(.x))
)
)
jsonlite::write_json(report,
file.path(output_dir, sprintf("qc_%s_%s.json", dataset, format(Sys.Date(), "%Y%m%d"))),
auto_unbox = TRUE, pretty = TRUE)
capture.output(print(result)) |>
writeLines(file.path(output_dir, sprintf("diffdf_%s.txt", dataset)))
report
}What R’s validation status actually is
A question that comes up constantly, and deserves a direct answer.
R itself is not “validated” — no software is validated in the abstract. Validation is something an organisation does to a system in its intended use. The R Foundation publishes a regulatory compliance document describing R’s development process, testing and quality practices, which supports an organisation’s own validation.
The FDA does not require SAS. The relevant guidance concerns data formats (SAS transport V5 for datasets) and the ability to reproduce analyses — not the software used. The FDA’s Statistical Software Clarifying Statement says explicitly that it does not require or endorse any particular software package.
Submissions using R have been accepted. The R Consortium’s R Submissions Working Group has completed several publicly documented pilot submissions to the FDA, including R-based ADaM datasets, TLFs and a Shiny application. Their materials are the best available reference.
What your organisation still has to do: qualify the environment, assess the packages, document the process, and produce the evidence above. That work is the same in any language.
Common mistakes
| Mistake | Consequence | Fix |
|---|---|---|
| QC programmer reads the production code | Not independent; misreadings propagate | Spec only |
Raising tolerance to hide a difference |
Real discrepancy suppressed | Investigate first |
| Comparing RTF files | Fails on timestamps, uninformative | Compare the numeric data |
| Unit tests only for the happy path | Edge cases break in production | Test empty, NA, boundaries |
| No traceability to requirements | Cannot demonstrate coverage | Requirement IDs in test names |
Treating riskmetric as validation |
Insufficient evidence | It is a screening tool |
| No record of which version produced what | Unanswerable audit question | Tag, renv.lock, provenance stamp |
| Differences resolved verbally | No evidence | Written resolution log |
Exercise 8.1 — Run and interpret a QC comparison
Two ADSL datasets differ. Write the comparison, then explain each difference type and how you would resolve it.
Show solution
library(diffdf); library(dplyr)
prod <- readRDS("data/adam/adsl.rds")
qc <- readRDS("data/adam_qc/adsl.rds")
result <- diffdf(
prod, qc,
keys = "USUBJID",
tolerance = 1e-8,
file = "output/qc/diffdf_adsl.txt"
)Differences found between the objects!
Not All Variables Compared Equal
Variable No of Differences
TRTDURD 3
AGEGR1 12
DCSREAS 1
Variables not in COMPARE
VARIABLE
RANDFL
Not all rows in BASE are in COMPARE
USUBJID
01-718-1427Difference 1 — TRTDURD differs for 3 subjects
prod |>
select(USUBJID, TRTSDT, TRTEDT, TRTDURD) |>
inner_join(select(qc, USUBJID, TRTDURD_QC = TRTDURD), by = "USUBJID") |>
filter(TRTDURD != TRTDURD_QC)
#> USUBJID TRTSDT TRTEDT TRTDURD TRTDURD_QC
#> 1 01-701-1015 2026-03-15 2026-09-12 182 183182 vs 183 on every affected subject — a systematic off-by-one. One programmer used TRTEDT - TRTSDT, the other TRTEDT - TRTSDT + 1.
Resolution. Check the specification. ADaM defines TRTDURD as TRTEDT - TRTSDT + 1 (treatment duration in days, inclusive of both endpoints). QC is correct; production must be fixed. Add a unit test:
test_that("REQ-018: TRTDURD is inclusive of both endpoints", {
expect_equal(compute_trtdurd(as.Date("2026-03-15"), as.Date("2026-03-15")), 1)
expect_equal(compute_trtdurd(as.Date("2026-03-15"), as.Date("2026-03-16")), 2)
})Difference 2 — AGEGR1 differs for 12 subjects
prod |>
select(USUBJID, AGE, AGEGR1) |>
inner_join(select(qc, USUBJID, AGEGR1_QC = AGEGR1), by = "USUBJID") |>
filter(AGEGR1 != AGEGR1_QC) |>
count(AGE, AGEGR1, AGEGR1_QC)
#> AGE AGEGR1 AGEGR1_QC n
#> 1 65 <65 65-80 8
#> 2 80 65-80 >80 4Every difference is exactly at a boundary. One programmer used < and the other <=.
Resolution. The specification says “65-80” — read literally that is 65 <= AGE <= 80, making QC correct. But this is exactly the ambiguity that double programming exists to find: “65-80” and “≥65 to <80” are both plausible readings of a range written that way.
The right resolution is not to pick one — it is to get the specification amended to state the boundaries unambiguously (“≥65 and ≤80”), then fix whichever program is wrong. Document the ambiguity and the amendment; it will recur on the next study otherwise.
Difference 3 — DCSREAS differs for 1 subject
prod |> filter(USUBJID == "01-704-1266") |> pull(DCSREAS)
#> [1] "ADVERSE EVENT"
qc |> filter(USUBJID == "01-704-1266") |> pull(DCSREAS)
#> [1] "Adverse Event"A case difference. One programmer applied str_to_upper(), the other did not.
Resolution. Check the spec and the controlled terminology. If DCSREAS is free text taken from DSTERM, it should be preserved as collected — production is wrong to upper-case it. If it is coded, the codelist decides. Either way, check_ct_data() from lesson 5 should have caught this, which suggests the codelist is missing from the specification.
Difference 4 — RANDFL present in production only
Production derived a variable QC did not. Either QC missed a specified variable, or production added an unspecified one.
Resolution. drop_unspec_vars() in the production program would have prevented this if RANDFL is not in the spec. If it is in the spec, QC has a gap. Check the spec first.
Difference 5 — one subject in production only
The most serious difference. A subject present in one dataset and not the other means the population definitions diverge.
setdiff(prod$USUBJID, qc$USUBJID)
#> [1] "01-718-1427"
dm |> filter(USUBJID == "01-718-1427") |> select(USUBJID, ARM, ARMCD, ACTARM)
#> USUBJID ARM ARMCD ACTARM
#> 1 01-718-1427 Screen Failure Scrnfail <NA>A screen failure. One programmer included screen failures in ADSL, the other excluded them.
Resolution. This is a specification question, and both are defensible — ADaM permits either. The specification must state it, and the define.xml must describe it. Resolve by amending the spec, then aligning both programs.
The resolution log
Every difference needs a written record:
| ID | Variable | Subjects | Cause | Resolution | Who | Date |
|---|---|---|---|---|---|---|
| D01 | TRTDURD | 3 | Off-by-one; spec is inclusive | Production corrected | RG | 2026-07-28 |
| D02 | AGEGR1 | 12 | Spec ambiguous at boundaries | Spec amended to “≥65 and ≤80”; production corrected | RG/JS | 2026-07-29 |
| D03 | DCSREAS | 1 | Case; spec has no codelist | Codelist added to spec; production corrected | RG | 2026-07-29 |
| D04 | RANDFL | all | Not in spec | Added to spec; QC corrected | JS | 2026-07-29 |
| D05 | (row) | 1 | Screen failure inclusion undefined | Spec amended to exclude; production corrected | RG/JS | 2026-07-30 |
Exercise 8.2 — Test suite for a derivation
Write a complete test suite for derive_agegr1(age, breaks, labels), including the boundary cases, and produce a traceability report linking tests to requirements.
Show solution
The function, with boundaries made explicit:
# R/derivations.R
#' Derive an age group category
#'
#' @param age Numeric vector of ages in years.
#' @param breaks Numeric vector of lower bounds for each group, ascending.
#' Each group is `[break, next_break)` — lower bound inclusive, upper
#' exclusive.
#' @param labels Character vector of labels, one per group.
#' @param missing_label Label for missing ages. Use `NA_character_` to leave
#' them missing.
#' @return A character vector the same length as `age`.
#' @export
derive_agegr1 <- function(age,
breaks = c(-Inf, 18, 65, Inf),
labels = c("<18", "18-64", ">=65"),
missing_label = "Missing") {
if (length(labels) != length(breaks) - 1) {
cli::cli_abort(c(
"{.arg labels} must have one fewer element than {.arg breaks}.",
"x" = "Got {length(breaks)} break{?s} and {length(labels)} label{?s}."
))
}
if (is.unsorted(breaks, strictly = TRUE)) {
cli::cli_abort("{.arg breaks} must be strictly increasing.")
}
if (!is.numeric(age)) {
cli::cli_abort("{.arg age} must be numeric, not {.obj_type_friendly {age}}.")
}
out <- as.character(
cut(age, breaks = breaks, labels = labels, right = FALSE)
)
out[is.na(age)] <- missing_label
out
}The test suite:
# tests/testthat/test-derive-agegr1.R
test_that("REQ-021: ages are assigned to the correct group", {
expect_equal(derive_agegr1(10), "<18")
expect_equal(derive_agegr1(30), "18-64")
expect_equal(derive_agegr1(70), ">=65")
})
test_that("REQ-021: lower bounds are inclusive and upper bounds exclusive", {
# This is the requirement that double programming most often disagrees on
expect_equal(derive_agegr1(17), "<18") # just below
expect_equal(derive_agegr1(18), "18-64") # exactly on the boundary
expect_equal(derive_agegr1(64), "18-64") # just below
expect_equal(derive_agegr1(65), ">=65") # exactly on the boundary
})
test_that("REQ-021: non-integer ages are handled at boundaries", {
expect_equal(derive_agegr1(17.9), "<18")
expect_equal(derive_agegr1(18.0), "18-64")
expect_equal(derive_agegr1(64.9), "18-64")
expect_equal(derive_agegr1(65.0), ">=65")
})
test_that("REQ-022: missing ages are labelled Missing", {
expect_equal(derive_agegr1(NA_real_), "Missing")
expect_equal(derive_agegr1(c(30, NA, 70)), c("18-64", "Missing", ">=65"))
})
test_that("REQ-022: the missing label is configurable", {
expect_equal(derive_agegr1(NA_real_, missing_label = "Unknown"), "Unknown")
expect_true(is.na(derive_agegr1(NA_real_, missing_label = NA_character_)))
})
test_that("REQ-023: custom breaks and labels are honoured", {
out <- derive_agegr1(
c(50, 70, 85),
breaks = c(-Inf, 65, 80, Inf),
labels = c("<65", "65-79", ">=80")
)
expect_equal(out, c("<65", "65-79", ">=80"))
})
test_that("the function is vectorised and preserves length and order", {
ages <- c(10, 30, 70, NA, 18, 65)
out <- derive_agegr1(ages)
expect_length(out, 6)
expect_type(out, "character")
expect_equal(out, c("<18", "18-64", ">=65", "Missing", "18-64", ">=65"))
})
test_that("empty input returns empty output", {
out <- derive_agegr1(numeric(0))
expect_length(out, 0)
expect_type(out, "character")
})
test_that("extreme values are assigned to the outer groups", {
expect_equal(derive_agegr1(0), "<18")
expect_equal(derive_agegr1(120), ">=65")
expect_equal(derive_agegr1(-1), "<18") # implausible but must not error
})
test_that("mismatched breaks and labels are rejected", {
expect_error(
derive_agegr1(30, breaks = c(-Inf, 18, 65, Inf), labels = c("a", "b")),
"one fewer element"
)
})
test_that("unsorted breaks are rejected", {
expect_error(
derive_agegr1(30, breaks = c(-Inf, 65, 18, Inf), labels = c("a", "b", "c")),
"strictly increasing"
)
})
test_that("non-numeric input is rejected", {
expect_error(derive_agegr1("30"), "must be numeric")
expect_error(derive_agegr1(factor(30)), "must be numeric")
})Traceability:
# tests/testthat/test-traceability.R
test_that("every requirement has at least one test", {
requirements <- readr::read_csv(testthat::test_path("..", "..",
"docs", "requirements.csv"),
show_col_types = FALSE)
test_files <- list.files(testthat::test_path(), pattern = "^test-",
full.names = TRUE)
all_tests <- unlist(lapply(test_files, function(f) {
grep('test_that\\("', readLines(f), value = TRUE)
}))
tested <- unique(na.omit(stringr::str_extract(all_tests, "REQ-\\d+")))
untested <- setdiff(requirements$id, tested)
expect_equal(
untested, character(0),
info = paste("Requirements with no test:", paste(untested, collapse = ", "))
)
})The report:
traceability_report <- function() {
reqs <- readr::read_csv("docs/requirements.csv", show_col_types = FALSE)
tests <- purrr::map(list.files("tests/testthat", "^test-", full.names = TRUE),
function(f) {
lines <- grep('test_that\\("', readLines(f), value = TRUE)
tibble::tibble(
file = basename(f),
test = stringr::str_match(lines, 'test_that\\("(.*?)"')[, 2]
)
}) |>
purrr::list_rbind() |>
mutate(id = stringr::str_extract(test, "REQ-\\d+"))
reqs |>
left_join(summarise(filter(tests, !is.na(id)),
n_tests = n(),
test_files = paste(unique(file), collapse = ", "),
.by = id),
by = "id") |>
mutate(n_tests = coalesce(n_tests, 0L),
status = if_else(n_tests > 0, "COVERED", "NOT TESTED"))
}
traceability_report()
#> # A tibble: 4 x 5
#> id description n_tests test_files status
#> <chr> <chr> <int> <chr> <chr>
#> 1 REQ-021 Age groups <18, 18-64, >=65 3 test-derive-agegr1.R COVERED
#> 2 REQ-022 Missing age labelled Missing 2 test-derive-agegr1.R COVERED
#> 3 REQ-023 Age groups configurable per study 1 test-derive-agegr1.R COVERED
#> 4 REQ-024 Age group order matches display order 0 NA NOT TESTEDThe boundary tests are the ones that earn their keep. derive_agegr1(65) is exactly the value where two independent implementations diverge, and the test makes the convention explicit in a form a reviewer can read without opening the code.
Recap
- Validation is system-level; verification is output-level; testing is code-level
- Independent double programming means the QC programmer works from the spec, not the code
diffdfwith the rightkeys; never raisetoleranceto hide a difference- Compare computed numbers, not RTF files
- Boundary and edge-case tests are where independent implementations diverge
- Requirement IDs in test names make traceability a
grep riskmetricscreens packages; it is not a validation- Most double-programming differences are resolved by amending the specification
Next: Define.xml preparation.