Skip to contents

Downloading POLIS data

Source: vignettes/downloading-polis-data.qmd

Note

The download calls in this vignette talk to the live POLIS service and are shown unevaluated — they need a valid POLIS_API_KEY and network access. The only chunk that runs at build time is the table catalogue, which is static package data. Copy the calls into a session with a key set to try them.

What get_polis_data() is for

POLIS exposes its data through an OData API, but three quirks make a naive OData client silently lose rows. polished::get_polis_data() is a single entry point that works around all three, caches results on disk, and resumes cleanly if a pull is interrupted.

The three traps it handles for you:

  • Date filters are year-aligned only. POLIS honours a field le YYYY-12-31 bound but returns zero rows for any sub-year le. get_polis_data() aligns your min_date / max_date to year boundaries before building a filter, so a mid-year range never silently empties the result.
  • $skip is rejected and there is no @odata.nextLink. POLIS caps $top at 2000 rows and refuses the standard OData paging mechanism. The only way past 2000 rows is Id-range pagination — …&$orderby=Id&$top=2000&$filter=… and Id gt <last> — which the function does for you, one page at a time.
  • The clinical date columns are sparsely populated. Columns like CaseDate and VirusDate are NULL for many historical records, so filtering on them drops rows. The function instead filters on each table’s update column (LastUpdateDate / UpdatedDate / Start / PublishDate), which probes confirm is fully populated.

Quick start

Set your key once, then pull a table. By default nothing is returned — the data lands on disk and you read it when you need it.

Sys.setenv(POLIS_API_KEY = "your-key") # or set it in .Renviron

# Pull the immunization (`im`) table; the data is written to the per-user
# cache and nothing is returned
polished::get_polis_data(tables = "im")

# Read the table back from disk when you need it (written under its raw_* name)
cache <- tools::R_user_dir("polished", which = "cache")
im <- readRDS(file.path(cache, "raw_im.rds"))

The table catalogue

tables accepts any of the names in polis_tables_mapping, the static catalogue the function ships with. Passing tables = NULL (the default) downloads them all. This is the one chunk in the vignette that runs at build time, because it is just package data:

polished::polis_tables_mapping
#>                   table_name                endpoint     date_field
#> 1                      virus                   Virus    UpdatedDate
#> 2                       case                    Case LastUpdateDate
#> 3             human_specimen             LabSpecimen LastUpdateDate
#> 4       environmental_sample               EnvSample LastUpdateDate
#> 5                   activity                Activity LastUpdateDate
#> 6               sub_activity             SubActivity    UpdatedDate
#> 7                       lqas                    Lqas          Start
#> 8                         im                      Im    PublishDate
#> 9        historized_synonyms      HistorizedSynonyms LastUpdateDate
#> 10 historized_geoplace_names HistorizedGeoplaceNames LastUpdateDate
#> 11                population              Population           <NA>
#>                        file_stem
#> 1                      raw_virus
#> 2                        raw_afp
#> 3                   raw_hum_spec
#> 4                         raw_es
#> 5                   raw_activity
#> 6               raw_sub_activity
#> 7                       raw_lqas
#> 8                         raw_im
#> 9        raw_historized_synonyms
#> 10 raw_historized_geoplace_names
#> 11                raw_population

Each row records the short table_name you pass to tables =, the OData endpoint it maps to, the date_field used for both filtering and the duplicate-resolution tiebreak, and the file_stem the table is written under on disk. That stem is the raw_* name the cleaning pipeline reads (e.g. case is saved as raw_afp), so the download and cleaning halves share one naming convention. An unknown name aborts with the list of valid ones, so a typo fails fast rather than fetching nothing.

Choosing what to fetch

Four arguments narrow the pull:

Argument Effect
min_date / max_date the date range, aligned to whole years (default 2000-01-01 to today)
region a WHO region filter ("Global", "AFRO", "EMRO", …)
country_code an ISO3 code (e.g. "NGA") added as an exact-match clause 
# Nigeria only, 2018 through last year, AFRO region
polished::get_polis_data(
  tables = "case",
  min_date = "2018-01-01",
  max_date = "2024-12-31",
  region = "AFRO",
  country_code = "NGA"
)

Because date bounds are year-aligned, min_date = "2018-06-30" fetches from 2018-01-01 regardless — the function never returns fewer rows than the year covering your bound.

Where the data lives, and how resume works

Every table is fetched into a per-year part file, with a tiny metadata sidecar, before being merged into one canonical file written under the table’s raw_* stem:

<polis_folder>/
├── raw_im.rds                  # canonical, merged file you read
└── .parts/raw_im/
    ├── year_2023.rds           # one part per calendar year
    ├── year_2023.meta.rds      # row count + Id range sidecar
    └── year_2024.rds

The part files are the resume marker. Each year is an independent Id-range walk that flushes to its part after every 2000-row batch, so if a run dies mid-pull — a dropped connection, a Ctrl-C, an OOM — the next call picks up at Id gt max(Id) for each year. Worst-case lost work is the single in-flight batch.

Files written by older versions under the bare table name (e.g. case.rds, .parts/case/) are renamed to their raw_* stem in place on the next run, so an existing cache is reused rather than re-downloaded.

polis_folder defaults to tools::R_user_dir("polished", which = "cache"), the standard per-user cache location, so incremental updates persist across sessions without you choosing a path. Pass an explicit folder to keep data alongside a project instead:

polished::get_polis_data(tables = "im", polis_folder = "data/polis")

Incremental updates

Re-running the same call is the update path. The function asks POLIS for the current row count, compares it against what the part files already hold, and:

  • if the cache already covers the count, it skips the pull entirely (just re-merging parts to the canonical file); otherwise
  • it walks each year from its last cached Id forward, fetching only the new rows.

So a daily cron that calls get_polis_data(tables = "case") downloads the full history once and only the delta thereafter.

To force a clean re-pull instead of resuming, pass force = TRUE (this deletes the table’s parts and canonical file first).

By default (prune_parts = TRUE) the per-year parts are deleted once the canonical file is written and verified; the next run rebuilds them from the canonical, which keeps the parts free of stale cross-year duplicates and keeps the “already up to date” short-circuit honest. Incremental resume still works. Pass prune_parts = FALSE to keep the parts on disk for the fastest possible resume (no re-split next run).

Going faster with parallel workers

Each calendar year is an independent walk, so years can be fetched concurrently. workers > 1 dispatches them across a parallel::makePSOCKcluster() — the same transport on Windows, macOS, and Linux — while a single live progress bar polls the part files so you watch rows accumulate across workers in real time:

polished::get_polis_data(
  tables = "virus",
  workers = parallel::detectCores() - 1L
)

Important

PSOCK workers start fresh R sessions and load polished with library(), so the package must be installed for parallel mode — a devtools::load_all() session is not enough. With workers = 1L (the default) a single sequential loop drives the bar per batch and has no such requirement.

What you get back

Nothing — and that’s the point. get_polis_data() is called purely for its side effect of writing each table to disk, and returns NULL invisibly. A single POLIS table can run to millions of rows, so returning that into memory on every call would be a footgun; instead the data stays on disk and you read back only the table you need, only when you need it.

Each table lands at <polis_folder>/<file_stem>.<ext> (the raw_* name from the catalogue). Read one back with the matching reader for your output_format:

# default per-user cache location
cache <- tools::R_user_dir("polished", which = "cache")

case <- readRDS(file.path(cache, "raw_afp.rds")) # `case` is saved as raw_afp
# or, for other output formats:
# arrow::read_parquet(file.path(cache, "raw_afp.parquet"))

Because the value is invisible, a bare get_polis_data(...) at the console prints nothing — assign the call if you want the paths.

Output formats

output_format controls how the canonical file is written: "rds" (default), "rda", "csv", "parquet", or "qs2". The parquet and qs2 formats need the arrow and qs2 packages respectively.

polished::get_polis_data(tables = "case", output_format = "parquet")

Completeness verification

POLIS occasionally truncates a query under load and returns a partial page even when more rows exist. With auto_refetch = TRUE (the default), after each table finishes the function:

  1. uses the metadata sidecars to detect a gap cheaply (row-count and Id-range mismatch against POLIS’s reported @odata.count);
  2. only if a gap is found, issues a lightweight $select=Id probe to list the canonical Id set; and
  3. refetches any missing Ids via Id in (...) chunks and merges them in, de-duplicating by Id keeping the latest update date.

Set auto_refetch = FALSE to trust whatever is on disk and skip the check — for example in the smoke-test path, or when you know the pull completed cleanly and want to save the verification round-trip.

Other options worth knowing

Argument What it does
keep_archives when > 0, also writes a timestamped copy under archive/ on each save and prunes older copies beyond this many
prune_parts when TRUE (default), deletes the .parts/ resume cache after the canonical is written and verified; rebuilt from the canonical next run
log_file path to a per-batch .rds log of what was fetched, when, and how many rows
quiet suppresses headers, progress bars, and the info alerts
polis_api_key the key; defaults to Sys.getenv("POLIS_API_KEY")

From download to clean

get_polis_data() gets you a faithful local copy of POLIS; the cleaners take it from there. A typical flow pulls a table, then recovers any missing administrative geography from the EPID:

polished::get_polis_data(tables = "case")

cache <- tools::R_user_dir("polished", which = "cache")
cases <- readRDS(file.path(cache, "raw_afp.rds")) # `case` is saved as raw_afp

cleaned <- polished::impute_geo_from_epid(cases)
cleaned$qa # what was filled, and what was left unresolved

See the Recovering geography from EPIDs vignette for that second step in detail, the End-to-end pipeline article to go from a folder of raw_* files straight to polished_* outputs, and ?get_polis_data for the complete argument reference.