Scaffolding a pipeline project

A real POLIS analysis is more than one run_pipeline() call: it has raw downloads, processed shapefiles, WorldPop rasters, a population reconciliation, and the cleaned polished_* outputs — each with a conventional home on disk, and a cfg that ties them together. init_polis_pipeline() stamps out that whole project in one call: the directory layout, a wired .Rprofile, a .gitignore, and runnable 2a_download_data.R / 2b_process_data.R starter scripts.

It is the heavier sibling of init_polis_project():

	`init_polis_project()`	`init_polis_pipeline()`
Layout	generic `raw/processed/validation/cache/logs` zones	domain-numbered `01_data` (shapefiles / population / polis / vaccination), `02_scripts`, `03_outputs`
Writes a `.Rprofile`?	no	yes — the `cfg` manifest
Writes starter scripts?	no	yes — download + process
Use it when	you just need tidy zones for one stream	you want the full reproducible download → clean pipeline

One call

root <- file.path(tempdir(), "polio_pipeline")

init_polis_pipeline(
  root,
  regions    = c("EMRO", "AFRO"), # WHO regions the pipeline is scoped to
  start_year = 2015,
  pop_years  = 2010:2027,
  pop_source = "reconciled"       # "reconciled" | "polis" | "worldpop"
)

The layout it produces

cat(sub(root, "<project>", list.dirs(root), fixed = TRUE), sep = "\n")
#> <project>
#> <project>/01_data
#> <project>/01_data/1a_shapefiles
#> <project>/01_data/1a_shapefiles/processed
#> <project>/01_data/1a_shapefiles/raw
#> <project>/01_data/1b_population
#> <project>/01_data/1b_population/polis_pop
#> <project>/01_data/1b_population/polis_pop/processed
#> <project>/01_data/1b_population/polis_pop/raw
#> <project>/01_data/1b_population/worldpop
#> <project>/01_data/1b_population/worldpop/processed
#> <project>/01_data/1b_population/worldpop/raw
#> <project>/01_data/1b_population/worldpop/raw/all
#> <project>/01_data/1b_population/worldpop/raw/u15
#> <project>/01_data/1b_population/worldpop/raw/u5
#> <project>/01_data/1c_polis
#> <project>/01_data/1c_polis/processed
#> <project>/01_data/1c_polis/processed/cache
#> <project>/01_data/1c_polis/processed/checks
#> <project>/01_data/1c_polis/processed/data
#> <project>/01_data/1c_polis/raw
#> <project>/01_data/1d_vaccination
#> <project>/01_data/1d_vaccination/processed
#> <project>/01_data/1d_vaccination/raw
#> <project>/02_scripts
#> <project>/03_outputs
#> <project>/03_outputs/plots
#> <project>/03_outputs/tables

Each 01_data domain has a raw/ (inputs) and processed/ (derived) half:

Domain	Holds	Produced by
`1a_shapefiles`	WHO polio GDB layers → `spatial_global_adm{0,1,2}.qs2`	`process_spatial()` (2b)
`1b_population/worldpop`	annual rasters → extracted adm2 tables	2a download + 2b extract
`1b_population/polis_pop`	`raw_population.qs2` → cleaned denominators	`get_polis_data()` + `clean_pop()`
`1c_polis`	`raw_.qs2` → `polished_.qs2` (+ `checks/`, `cache/`)	`get_polis_data()` + `run_pipeline()`
`1d_vaccination`	coverage rasters (optional downstream models)	your own scripts

Pass domains = to scaffold a subset (e.g. c("shapefiles", "polis")).

The wired `.Rprofile`

The generated .Rprofile is the project manifest — the data-directory roots are defined once as the single source of truth, and cfg is built from them, so the scripts and the cleaners share exactly the same paths:

cat(readLines(file.path(root, ".Rprofile")), sep = "\n")
#> # .Rprofile -- generated by polished::init_polis_pipeline()
#> # The data-directory roots below are the single source of truth shared by `cfg`
#> # and the 02_scripts/. Edit `regions` / `start_year` / `pop_years` /
#> # `pop_source` to taste.
#> 
#> if (file.exists("renv/activate.R")) source("renv/activate.R")
#> 
#> if (
#>   requireNamespace("polished", quietly = TRUE) &&
#>     requireNamespace("here", quietly = TRUE)
#> ) {
#>   shapes_raw <- here::here("01_data", "1a_shapefiles", "raw")
#>   shapes_proc <- here::here("01_data", "1a_shapefiles", "processed")
#>   wp_raw <- here::here("01_data", "1b_population", "worldpop", "raw")
#>   wp_proc <- here::here("01_data", "1b_population", "worldpop", "processed")
#>   pop_raw <- here::here("01_data", "1b_population", "polis_pop", "raw")
#>   pop_proc <- here::here("01_data", "1b_population", "polis_pop", "processed")
#>   polis_raw <- here::here("01_data", "1c_polis", "raw")
#>   polis_proc <- here::here("01_data", "1c_polis", "processed")
#> 
#>   cfg <- polished::polis_config(
#>     start_year = 2015,
#>     regions = c("EMRO", "AFRO"),
#>     pop_years = 2010:2027,
#>     pop_source = "reconciled",
#>     shape = file.path(shapes_proc, "spatial_global_adm2.qs2"),
#>     # clean_pop() consumes the adm2-by-year tables 2b extracts from the
#>     # WorldPop rasters (it also accepts raster directories directly).
#>     worldpop = list(
#>       all = file.path(wp_proc, "global_worldpop_total_pop.qs2"),
#>       u5 = file.path(wp_proc, "global_worldpop_u5_pop.qs2"),
#>       u15 = file.path(wp_proc, "global_worldpop_u15_pop.qs2")
#>     ),
#>     inputs = list(
#>       afp = file.path(polis_raw, "raw_afp.qs2"),
#>       hum_spec = file.path(polis_raw, "raw_hum_spec.qs2"),
#>       es = file.path(polis_raw, "raw_es.qs2"),
#>       activity = file.path(polis_raw, "raw_activity.qs2"),
#>       subactivity = file.path(polis_raw, "raw_sub_activity.qs2"),
#>       lqas = file.path(polis_raw, "raw_lqas.qs2"),
#>       im = file.path(polis_raw, "raw_im.qs2"),
#>       population = file.path(pop_raw, "raw_population.qs2")
#>     ),
#>     output_dir = polis_proc,
#>     cache_dir = file.path(polis_proc, "cache")
#>   )
#> }

The four knobs you passed to init_polis_pipeline() are interpolated into the polis_config() call, so editing them later is just editing the .Rprofile:

regions — scopes the surveillance streams (AFP/ES/SIA/…). Population is foundational and stays global regardless.
start_year — earliest onset/collection year retained.
pop_years — the population/WorldPop window.
pop_source — which value clean_pop() chooses as <age>_pop: "reconciled" (POLIS, with WorldPop fallback on missing/implausible values), "polis" (POLIS only), or "worldpop" (WorldPop only). All three of _pop / _pop_polis / _pop_wp are always kept, so this only changes the default column — you can pick a different denominator downstream without re-running.

Because cfg is built outside any if (interactive()) guard (and behind a requireNamespace() check), it is available both in RStudio and under Rscript, so the starter scripts run headless.

The starter scripts

list.files(file.path(root, "02_scripts"))
#> [1] "2a_download_data.R" "2b_process_data.R"

2a_download_data.R pulls every input: the POLIS population reference, the seven POLIS surveillance tables (get_polis_data()), and the WorldPop rasters (sntutils::download_worldpop*). Only missing files are (re)fetched. The raw WHO polio GDB shapefile layers are the one thing it does not download — drop those into 01_data/1a_shapefiles/raw/ yourself.

2b_process_data.R runs in three sections, in order:

Spatial — process_spatial() cleans the GDB layers to spatial_global_adm2.qs2 (everything else keys off it).
WorldPop — sums the annual rasters to adm2 tables (cached on disk; re-extracted only if missing), which clean_pop() then consumes.
POLIS — run_pipeline(cfg = cfg) cleans every stream (AFP / ES / SIA / LQAS / IM / virus / indicators and population via clean_pop()) and writes polished_* outputs plus checks_* workbooks.

Running it end to end

# 0. drop the WHO polio GDB layers into 01_data/1a_shapefiles/raw/
# 1. download every input
source("02_scripts/2a_download_data.R")
# 2. process: shapefile -> worldpop extract -> run_pipeline over all streams
source("02_scripts/2b_process_data.R")
# 3. read a country / period slice of the cleaned outputs back
afg <- load_polished(country = "AFG")
names(afg)

load_polished() defaults output_dir to the active config, so once the .Rprofile has run no path argument is needed — just the filter.

Reproducible package versions with `renv`

Pass renv = TRUE to wire the project for renv, so everyone installs the same package versions:

init_polis_pipeline("my_project", regions = "EMRO", renv = TRUE)

That runs renv::scaffold() — writing renv/activate.R, a starter renv.lock, and a single clean autoloader line in the generated .Rprofile (source("renv/activate.R"), no duplicate). The reproducibility loop is then:

install the packages the scripts need (polished, sntutils, sf, qs2, terra, exactextractr, …);
renv::snapshot() — pins those exact versions into renv.lock (renv finds them from the pkg::fun calls in 2a / 2b and the .Rprofile);
commit renv.lock;
a collaborator runs renv::restore() to reproduce the library exactly.

With renv = FALSE (the default) the .Rprofile still carries a guarded autoloader — if (file.exists("renv/activate.R")) source("renv/activate.R") — so the project is renv-ready: run renv::init() yourself later and nothing else in the scaffold changes.

Conventions baked in

Idempotent. Re-running never clobbers an edited .Rprofile or script; existing files are kept unless you pass overwrite = TRUE. Directory creation is always safe to repeat.
.here sentinel. A .here marker is written so here::here() in the .Rprofile resolves to the project root even before it is a git repo or RStudio project.
Sensitive raw data stays out of git. The generated .gitignore excludes 01_data/**/raw/ — raw POLIS tables are individual-level line-lists (and large), so they should not be committed. Re-download them with 2a instead.
One source of truth for paths. The .Rprofile defines the roots; the scripts and cfg reference them, so there is never a second, drifting copy.

For a lighter, single-stream workspace (no .Rprofile, no scripts) see init_polis_project() and the End-to-end pipeline article.