Clean POLIS population and (optionally) reconcile against WorldPop

Builds adm0/adm1/adm2 population denominators (under-5, under-15, all-ages) from the raw POLIS Population table. Used standalone it cleans POLIS by itself – collapses duplicate (place, year) rows to their median, drops zeros/blanks to missing, flags values that jump from a district's own history, and fills gaps from a district -> province -> country ladder. Given worldpop it adds a cross-source layer: each POLIS value is checked against WorldPop, and one that is missing or implausible is replaced by a substitution ladder that keeps the district on its own population level.

Usage

clean_pop(
  population,
  cfg = polis_active_config(),
  shape = NULL,
  worldpop = NULL,
  years = 2010:2027,
  thresholds = list(ratio_lo = 1/3, ratio_hi = 3, mad_k = 5, min_votes = 1L, dens_lo = 5,
    share_lo = 0.2, share_hi = 0.7, min_level_years = 2L),
  reference_date = Sys.Date(),
  pop_source = c("reconciled", "polis", "worldpop"),
  verbose = TRUE
)

Arguments

population

A raw POLIS Population data frame (columns PlaceId, PlaceDisplayName, Year, AgeGroupName, Value), or a path to one.

cfg

A polis_config() object; defaults to polis_active_config(). Only its presence is required – clean_pop reads no scoping from it (pop is foundational/global), but it keeps the cleaner signature uniform.

shape

Optional already-processed district shape (an sf polygon layer or its long ADM2 attribute table, or a path to one). Supplies the adm0/adm1 parents POLIS lacks, the boundary-validity windows used for the roll-ups, and the universe of districts. With no shape the output is keyed on adm2_guid only and cannot be rolled up. Default NULL.

worldpop

Optional named list with elements all, u5, u15. Each is either a directory of annual WorldPop GeoTIFFs (one per year, the year in the file name; zonal-summed to the shape via terra + exactextractr – both optional Suggests), or a pre-extracted adm2-by-year table (data frame or path) carrying adm2_guid, year and a population column. NULL (default) runs the POLIS-only path.

years

Calendar years to keep (POLIS carries 1990-2034 incl. projections). Default 2010:2027.

thresholds

Named list of the implausibility tunables:

ratio_lo, ratio_hi: a POLIS value below/above this fold of WorldPop is implausible (bad_vs_worldpop, and against the province's typical ratio for bad_vs_adm1).
mad_k: scaled-MAD distance from the district's own median (bad_vs_history) – the signal that catches a single bad year.
min_votes: how many signals must fire to call a value suspect.
dens_lo: people per km2 below which WORLDPOP is treated as the implausible source rather than the arbiter (wp_implausible). Needs a polygon shape; without one the signal never fires. Guards the case where WorldPop's raster allocation empties a district out and would otherwise convict a correct POLIS value.
share_lo, share_hi: bounds on the under-15 share of all-ages used when reporting age-band coherence.
min_level_years: how many trusted years a district needs before a POLIS:WorldPop level ratio is established from them. A level cannot be inferred from one observation, so below this count the district falls to raw WorldPop instead of being rescaled on a ratio it has no evidence for.

Default list(ratio_lo = 1/3, ratio_hi = 3, mad_k = 5, min_votes = 1L, dens_lo = 5, share_lo = 0.2, share_hi = 0.7, min_level_years = 2L). Supplying a partial list overrides only the keys given.

reference_date

Date treated as "today" when deciding which boundary versions are current for the orphan-GUID name crosswalk. Default Sys.Date().

pop_source

Which population to use as the chosen <age>_pop value (the denominator indicators read). One of:

"reconciled": (default) a trusted POLIS value, then the district's own interpolated series, then WorldPop rescaled onto the district's level, then raw WorldPop, then the district -> province -> country ladder. See the substitution ladder in Details.
"polis": the POLIS value, with interior gaps interpolated from the district's own series and the rest from the admin ladder; WorldPop is ignored even if supplied – the full POLIS population.
"worldpop": the WorldPop value, else a POLIS value, else the ladder. No levelling: this mode is asking for WorldPop's own numbers.

The output always keeps <age>_pop_polis and <age>_pop_wp alongside the chosen <age>_pop, so every source stays inspectable whatever the mode.

verbose

Emit cli progress headers. Default TRUE.

Value

A named list:

adm2: district x year, wide: the id columns plus, per age band (u5/u15/all), <age>_pop (chosen), <age>_pop_polis, <age>_pop_wp, <age>_pop_source (polis / polis_interp / worldpop_levelled / worldpop / district_trend / adm1 / adm0), <age>_pop_imputed, <age>_pop_level_ratio (the POLIS:WorldPop ratio used to put a substituted value on the district's level – NA when none was needed or none could be formed) and <age>_pop_frozen (the POLIS value repeated verbatim from the previous year). Plus age_order_bad and age_source_split (the bands do not share a level – non-empty only where no band had a ratio to lend). Restricted to the boundary valid each year (no double-counting versioned shapes).
adm1, adm0: province / country roll-ups (sums) of the nine pop columns. The per-district ratio and frozen flags are deliberately not rolled up: neither is meaningful summed across districts.
meta: a list (skipped by the file writer): audit (one row per district x year x age, with every signal flag – bad_vs_worldpop, bad_vs_history, bad_vs_adm1, wp_implausible, frozen, n_votes, polis_suspect), dup_conflicts, orphan_xwalk, params.

What goes wrong in POLIS population, and what catches it

Two jobs, deliberately separate. Detection decides whether a value is usable; substitution decides what replaces it when it is not. Conflating them is how a repaired value becomes a worse problem than the one it fixed (see Why substitution is level-matched below).

Fault	Origin	Handled by
Duplicate `(place, year)` rows, often conflicting	POLIS	median collapse; the conflicts are returned in `meta$dup_conflicts`
Zeros / blanks presented as real values	POLIS	dropped to `NA`
A single year wildly off the district's own series	POLIS	`bad_vs_history` (scaled MAD, `mad_k`)
A district carrying its PARENT's population – typically where an adm2 shares a name with its adm1	POLIS	`bad_vs_worldpop` + `bad_vs_adm1`
District GUIDs that have since changed boundary	POLIS	orphan name crosswalk; unresolved ones stay in `meta$orphan_xwalk`
u5 > u15 > all-ages ordering breaches	POLIS	`age_order_bad` + whole-set fallback
WorldPop emptying out over water and dense urban cores, so it convicts a POLIS value for being right	WorldPop	`wp_implausible` (density, `dens_lo`)
The three age bands landing on different sources, so they no longer describe one population	clean_pop	shared level ratio + `age_source_split`
A year repeated verbatim from the one before – a refresh that did not happen	POLIS	`<age>_pop_frozen` (flagged, never substituted)
POLIS coverage starting late / ending early, so head and tail years have no POLIS at all	POLIS	the substitution ladder

Two further properties are not faults but must be understood before the output is used. The POLIS-to-WorldPop gap is heterogeneous across districts – in Nigeria the ratio spans roughly 0.7 to 2.7 with a long tail – so no single national rescaling can reconcile them, which is why every correction here is per district. And POLIS tends to run high in aggregate (its Nigerian under-15 total implies a country larger than the UN estimate), so rates computed on it are correspondingly lower. Both are visible because <age>_pop_polis and <age>_pop_wp are always retained.

Why substitution is level-matched

The two sources disagree about a district's level, not just its value. So replacing a rejected POLIS year with WorldPop's raw number – the obvious move, and what this function used to do – turns a rejected year into a rejected level: the series then steps by the gap between two sources at whichever years happened to be rejected. That is a district-specific, year-specific discontinuity, and it is the single hardest kind for anything downstream to absorb, because a fitted model reads it as a real change in whatever the denominator feeds rather than as an artefact.

So a district never leaves its own level. The ladder, in order:

polis – a POLIS value that survived every signal.
polis_interp – an interior gap, linearly interpolated from the district's own trusted years. Never extrapolates.
worldpop_levelled – head/tail years, where there is no trusted value on one side and the shape has to come from WorldPop. Rescaled by <age>_pop_level_ratio, the POLIS:WorldPop ratio at the district's nearest trusted year. Nearest rather than an average because the two sources grow at different rates, and a summary ratio makes a projected year step down on a rising series.
worldpop – raw, and only for a district POLIS never usably described, which therefore has no level of its own to hold.
district_trend / adm1 / adm0 – the admin ladder, when there is no WorldPop either.

A rejected outlier is treated exactly as a missing value, so detection and substitution stay independent.

On Nigerian adm2 (774 districts, 2016-2026) the ladder takes district-years moving more than 25% year-on-year from 780 to 21, raw-WorldPop fallbacks from about 2,880 to 484, and districts whose age bands sit on different levels from 23 to 0.

Worked examples

EWEKORO (Ogun, Nigeria) – POLIS reports 131,998 under-15s for 2019 against roughly 39,000 either side. bad_vs_history rejects it; rung 2 interpolates 40,062 from the district's own 2018 and 2020. Its 2016, 2017 and 2026 have no POLIS at all and take rung 3.

KATSINA (Nigeria) – an adm2 sharing its name with its adm1, reporting 2.1-3.9M under-15s where the district holds roughly 265,000. bad_vs_worldpop and bad_vs_adm1 reject every year, so no trusted value exists to define a level and the district correctly falls to rung 4.

BAKASSI (Cross River, Nigeria) – the opposite case. WorldPop gives 82 under-15s over 26 km2, about 3 per km2, because the district is largely water. Without the density signal WorldPop would convict POLIS's 26,536; wp_implausible disqualifies WorldPop as the arbiter instead, POLIS is kept, and the level ratio is around 315.

Examples

pop_raw <- data.frame(
  PlaceId = "0cda1c45-9529-4188-aaaa-000000000001",
  PlaceDisplayName = "SOMEWHERE",
  Year = 2020, AgeGroupName = "0 to 15 years", Value = 1000,
  check.names = FALSE
)
res <- clean_pop(pop_raw, years = 2020, verbose = FALSE)
names(res)
#> [1] "adm0" "adm1" "adm2" "meta"