Skip to contents

Clean POLIS population and (optionally) reconcile against WorldPop

Source: R/clean_pop.R
clean_pop.Rd

Builds adm0/adm1/adm2 population denominators (under-5, under-15, all-ages) from the raw POLIS Population table. Reduces duplicate (place, year) rows to their median, treats zeros and blanks as missing, flags departures from district history and fills gaps using district or parent-area estimates. When worldpop is supplied, also compares the two sources and uses WorldPop in the replacement rules described below.

Usage

clean_pop(
  population,
  cfg = polis_active_config(),
  shape = NULL,
  worldpop = NULL,
  years = 2010:2027,
  thresholds = list(ratio_lo = 1/3, ratio_hi = 3, mad_k = 5, min_votes = 1L, dens_lo = 5,
    share_lo = 0.2, share_hi = 0.7, min_level_years = 2L),
  reference_date = .polis_reference_date(cfg),
  pop_source = c("reconciled", "polis", "worldpop"),
  verbose = TRUE
)

Arguments

population

A raw POLIS Population data frame (columns PlaceId, PlaceDisplayName, Year, AgeGroupName, Value), or a path to one.

cfg

A polis_config() object; defaults to polis_active_config(). Only its presence is required – clean_pop reads no scoping from it (pop is foundational/global), but it keeps the cleaner signature uniform.

shape

Optional already-processed district shape (an sf polygon layer or its long ADM2 attribute table, or a path to one). Supplies the adm0/adm1 parents POLIS lacks, the boundary-validity windows used for the roll-ups, and the universe of districts. With no shape the output is keyed on adm2_guid only and cannot be rolled up. Default NULL.

worldpop

Optional named list with elements all, u5, u15. Each is either a directory of annual WorldPop GeoTIFFs (one per year, the year in the file name; zonal-summed to the shape via terra + exactextractr – both optional Suggests), or a pre-extracted adm2-by-year table (data frame or path) carrying adm2_guid, year and a population column. NULL (default) runs the POLIS-only path.

years

Calendar years to keep (POLIS carries 1990-2034 incl. projections). Default 2010:2027.

thresholds

Named list of the implausibility tunables:

ratio_lo, ratio_hi

a POLIS value below/above this fold of WorldPop is implausible (bad_vs_worldpop, and against the province's typical ratio for bad_vs_adm1).

mad_k

scaled-MAD distance from the district's own median (bad_vs_history) – the signal that catches a single bad year.

min_votes

how many signals must fire to call a value suspect.

dens_lo

people per km2 below which WORLDPOP is treated as the implausible comparison source (wp_implausible). Needs a polygon shape; without one the signal never fires. Guards the case where a low WorldPop estimate would otherwise cause a POLIS value to be rejected.

share_lo, share_hi

bounds on the under-15 share of all-ages used when reporting age-band coherence.

min_level_years

how many trusted years a district needs before a POLIS:WorldPop level ratio is established from them. A level cannot be inferred from one observation, so below this count the district falls to raw WorldPop instead of being rescaled on a ratio it has no evidence for.

Default list(ratio_lo = 1/3, ratio_hi = 3, mad_k = 5, min_votes = 1L, dens_lo = 5, share_lo = 0.2, share_hi = 0.7, min_level_years = 2L). Supplying a partial list overrides only the keys given.

reference_date

Date treated as "today" when deciding which boundary versions are current for the orphan-GUID name crosswalk. Defaults to cfg$reference_date, or Sys.Date() when it is NULL.

pop_source

Which population to use as the chosen <age>_pop value (the denominator indicators read). One of:

"reconciled"

(default) a trusted POLIS value, then the district's own interpolated series, then WorldPop rescaled onto the district's level, then raw WorldPop, then the district -> province -> country ladder. See the substitution ladder in Details.

"polis"

the POLIS value, with interior gaps interpolated from the district's own series and the rest from the admin ladder; WorldPop is ignored even if supplied – the full POLIS population.

"worldpop"

the WorldPop value, else a POLIS value, else the ladder. No levelling: this mode is asking for WorldPop's own numbers.

The output always keeps <age>_pop_polis and <age>_pop_wp alongside the chosen <age>_pop, so every source stays inspectable whatever the mode.

verbose

Emit cli progress headers. Default TRUE.

Value

A named list:

adm2

district x year, wide: the id columns plus, per age band (u5/u15/all), <age>_pop (chosen), <age>_pop_polis, <age>_pop_wp, <age>_pop_source (polis / polis_interp / worldpop_levelled / worldpop / district_trend / adm1 / adm0), <age>_pop_imputed, <age>_pop_level_ratio (the POLIS:WorldPop ratio used to put a substituted value on the district's level – NA when none was needed or none could be formed) and <age>_pop_frozen (the POLIS value repeated verbatim from the previous year). Plus age_order_bad and age_source_split (the bands do not share a level – non-empty only where no band had a ratio to lend). Restricted to the boundary valid each year (no double-counting versioned shapes).

adm1, adm0

province / country roll-ups (sums) of the nine pop columns. The per-district ratio and frozen flags are deliberately not rolled up: neither is meaningful summed across districts.

meta

a list (skipped by the file writer): audit (one row per district x year x age, with every signal flag – bad_vs_worldpop, bad_vs_history, bad_vs_adm1, wp_implausible, frozen, n_votes, polis_suspect), dup_conflicts, orphan_xwalk, params.

Checks and replacement rules

Duplicate (place, year) values are reduced to their median; conflicting values are recorded in meta$dup_conflicts. Zeros and blanks become NA. The checks compare each district with its own history (bad_vs_history) and, when WorldPop is supplied, with WorldPop and province-level ratios (bad_vs_worldpop, bad_vs_adm1). Thresholds control which values are flagged.

Additional checks record unresolved district mappings (meta$orphan_xwalk), inconsistent age-band ordering (age_order_bad), age bands using different sources (age_source_split) and unchanged annual values (<age>_pop_frozen). Frozen values are flagged but are not replaced on that basis alone. With polygon boundaries, wp_implausible identifies WorldPop estimates below the configured population-density threshold.

Replacing missing or rejected values

The reconciled series uses the following sources in order. Interpolation and rescaling use the district's accepted POLIS values to reduce abrupt changes caused by switching between population sources.

  1. polis: a POLIS value accepted by the configured checks.

  2. polis_interp: linear interpolation between accepted POLIS years. This step does not extrapolate beyond those years.

  3. worldpop_levelled: WorldPop rescaled using the POLIS-to-WorldPop ratio at the nearest accepted year, when enough paired years are available.

  4. worldpop: the unscaled WorldPop estimate when no usable POLIS level ratio is available.

  5. district_trend / adm1 / adm0: estimates from district trends or parent administrative areas when the preceding sources are unavailable.

Missing and rejected values follow the same replacement rules. The output retains <age>_pop_polis and <age>_pop_wp alongside the selected population so users can compare sources. Passing the checks does not establish that an estimate is the true population.

See also

run_pipeline(), which runs this as the population stream; checks_pop(), which turns the result into a data-quality workbook.

Examples

pop_raw <- data.frame(
  PlaceId = "0cda1c45-9529-4188-aaaa-000000000001",
  PlaceDisplayName = "SOMEWHERE",
  Year = 2020, AgeGroupName = "0 to 15 years", Value = 1000,
  check.names = FALSE
)
res <- clean_pop(pop_raw, years = 2020, verbose = FALSE)
names(res)
#> [1] "adm0" "adm1" "adm2" "meta"