Getting started with spacescans

spacescans links time-referenced geolocation data to the spatial and contextual exposome. Its input is a cohort of location episodes: a participant, a location (longitude and latitude) and the first and last day spent there. For every location episode it computes area- or buffer-weighted exposures such as the Neighborhood Deprivation Index, walkability, noise, night-time lights or air pollution, or the distance to the nearest feature, and by default it returns one row per participant.

The R package is a client of the spacescans Python pipeline. Every call starts python -m spacescans.bridge in a separate process and exchanges JSON files and Parquet tables with it, so a crash or Ctrl-C in Python never takes your R session down. The code below does not run when this vignette is built, because it needs the Python environment and the exposure data; the two example scripts it shows are installed with the package and run by the package’s live tests.

1. Install the Python side once

library(spacescans)
ss_install()          # a private Python 3.12 environment, built with uv
ss_doctor()           # interpreter, versions, extras, folders, disk, data

ss_install() puts the environment under tools::R_user_dir("spacescans", "data") and never touches the Python on your PATH. TEMIS needs one more extra: ss_install(extras = "hdf4"). If your IT department provides an interpreter, point the package at it with ss_use_python() or the SPACESCANS_PYTHON environment variable. The supported platform is macOS 14 or later on Apple silicon.

2. Choose a data directory and get the data

No exposure data ships with spacescans. The catalog says where each dataset comes from, and ss_fetch() downloads what you need from the providers or from the project mirror, where the providers’ terms allow it.

ss_data_dir("~/ss-data")              # or set SPACESCANS_DATA_DIR
ss_catalog()                          # the variables, years, data and extras
ss_data_status("noise")
ss_fetch("noise", dry_run = TRUE)     # what would be downloaded (about 1.2 GB)
ss_fetch("noise")                     # resumable; shared files only once

Some provider datasets (for example EPA FAQSD) require you to accept their terms. Read them with ss_data_terms("faqsd"); an interactive session asks you, a script passes accept_terms =. Keep the data directory on a local disk: spacescans warns when it, the cache or a run directory is inside iCloud Drive, OneDrive, Dropbox or Box.

3. A minimal example

Three synthetic participants and the NPS noise model (Mennitt et al.), the catalog source whose files are all on the project mirror. The script is installed with the package: source(system.file("examples", "minimal-example.R", package = "spacescans")).

# A minimal spacescans example: three synthetic participants linked to the
# NPS noise model, the catalog source whose files are all on the project
# mirror. Before running it, once:
#   ss_install()              # the Python side
#   ss_data_dir("~/ss-data")  # where the exposure data goes
#   ss_fetch("noise")         # about 1.2 GB
# Then:
#   source(system.file("examples", "minimal-example.R", package = "spacescans"))
library(spacescans)

# Time-referenced geolocation data: one row per location episode. P2 moved
# once, so the three participants have four location episodes. The places
# are city centres, not people.
cohort <- data.frame(
  pid = c("P1", "P2", "P2", "P3"),
  startDate = as.Date(c("2015-01-01", "2014-03-01", "2016-07-01", "2017-01-01")),
  endDate = as.Date(c("2017-12-31", "2016-06-30", "2019-12-31", "2018-06-30")),
  longitude = c(-86.1581, -87.6298, -84.3880, -122.4194),
  latitude = c(39.7684, 41.8781, 33.7490, 37.7749)
)

if (!isTRUE(attr(ss_data_status("noise"), "ready"))) {
  stop('The noise data is not in ss_data_dir() yet: run ss_fetch("noise") first.')
}

# Participant level (the default): one row per participant, each value the
# mean over the participant's location episodes weighted by contributing days.
res <- ss_link(cohort, "noise")
print(res)
print(attr(res, "coverage"))      # linkage rates: the share of participants with a value
print(attr(res, "features"))      # what each column means

# Episode level: the cohort with the noise columns added, one row per episode.
episodes <- ss_link(cohort, "noise", output_level = "episode")
print(episodes)

# The two agree: noise is time-invariant, so every day of an episode
# contributes and P2's value is the day-weighted mean of its two episodes.
days <- as.numeric(episodes$endDate - episodes$startDate) + 1
p2 <- episodes$pid == "P2"
stopifnot(isTRUE(all.equal(res$l50dba_exi[res$pid == "P2"],
                           weighted.mean(episodes$l50dba_exi[p2], days[p2]))))

res has one row per participant (P1, P2, P3) and the columns pid, l50dba_exi, l50dba_imp and l50dba_nat; episodes is the cohort with the same three columns added, one row per location episode. The last line checks the participant-level rule on P2, who has two location episodes.

4. Output levels

ss_link() returns one row per participant unless you ask for output_level = "episode". For participant i and variable v the value is the mean of the participant’s episode values weighted by contributing days:

\[\bar{x}_{iv} = \frac{\sum_{e \in E_i} \sum_{t \in T_{ev}} d_{et}\, z_{etv}}{\sum_{e \in E_i} \sum_{t \in T_{ev}} d_{et}}\]

where \(E_i\) are the participant’s location episodes, \(z_{etv}\) is the spatial estimate (a buffer-weighted mean, or the distance to the nearest feature) for episode \(e\) in source period \(t\) (a year, two weeks or a day), \(T_{ev}\) are the source periods that overlap the episode and have an estimate, and \(d_{et}\) is the number of days of that overlap, first and last day included. A time-invariant source counts every day of the episode. Days are counted per variable, so a participant can have a value for one variable and not for another; a participant with no contributing day is kept, with a missing value. Where two location episodes of one participant overlap in time, each contributes its own days, so a calendar day both cover counts once for each location. Variables derived after aggregation (the FARA flags, the ACAG _nbm columns) are derived from the participant’s means; see ?ss_link.

	`"participant"` (default)	`"episode"`
rows	one per participant, in order of first appearance	one per location episode, in your order
columns	`pid` (your id’s type) and the exposures	your columns, unchanged, and the exposures
`attr(res, "coverage")`	share of participants with a value	share of episodes with a value

Both carry attr(res, "features") (what each column means), attr(res, "provenance") (versions, code fingerprint, data receipts, cache keys; no cohort values) and attr(res, "output_level"). ss_coverage() estimates before a run how many location episodes fall inside each variable’s years and region; it counts episodes at either level, because it looks only at dates and locations.

5. Link your cohort

A cohort has one row per location episode with the columns pid, startDate, endDate, longitude and latitude. Other names can be mapped with cols, and PATID is accepted for pid.

cohort <- data.frame(
  participant_id = c("A", "A", "B"),
  start = as.Date(c("2015-01-01", "2017-06-01", "2016-03-15")),
  end = as.Date(c("2017-05-31", "2019-12-31", "2019-12-31")),
  lon = c(-86.158, -86.131, -85.680),
  lat = c(39.768, 39.790, 40.193),
  age = c(54L, 56L, 71L)
)
cohort
#>   participant_id      start        end     lon    lat age
#> 1              A 2015-01-01 2017-05-31 -86.158 39.768  54
#> 2              A 2017-06-01 2019-12-31 -86.131 39.790  56
#> 3              B 2016-03-15 2019-12-31 -85.680 40.193  71

cols <- c(pid = "participant_id", startDate = "start", endDate = "end",
          longitude = "lon", latitude = "lat")
ss_coverage(cohort, c("ndi", "walkability"), cols = cols)

res <- ss_link(cohort, c("ndi", "walkability"), cols = cols)
res                       # pid, ndi, NatWalkInd: one row for A, one for B
ep <- ss_link(cohort, c("ndi", "walkability"), cols = cols, output_level = "episode")
ep                        # your columns, unchanged, plus ndi and NatWalkInd

What ss_link() does, in order:

checks the cohort in R and reports problems by row number, never by value;
writes only the five required columns to a private run directory (mode 0700, under tools::R_user_dir("spacescans", "cache")/runs); age and any other column never leave R;
checks that the data and the Python extras are in place, and otherwise tells you the exact ss_fetch() or ss_install() call to run;
runs the pipeline, showing progress; Ctrl-C stops it cleanly;
returns the participant table, or joins the exposures back onto your data frame by row (keeping its types and attributes), and warns when a variable has a value for fewer than 90% of the rows;
deletes the run directory (unless keep = TRUE). Should R be killed before it can, the next session’s package load (or ss_doctor(), ss_cache_clear()) deletes it, since it holds the five cohort columns.

The C3 weights (which units each buffer around a location overlaps) are cached under tools::R_user_dir("spacescans", "cache"), so linking the same cohort again is fast. They can reveal where people were: see ss_cache_info() and clear them with ss_cache_clear() when a project ends.

6. Long runs as background jobs

job <- ss_link(cohort, c("ndi", "noise", "vnl"), cols = cols,
               wait = FALSE, run_dir = "~/ss-runs/noise-vnl")
ss_job_status(job)
res <- ss_job_result(job)          # waits, builds the result as above, removes the run directory

ss_job_result() removes the run directory once it has the result (pass keep = TRUE to ss_link() to keep it). A job started with an explicit run_dir keeps running when R exits; in a new R session, instead of the ss_job_result() call above, find it again from its run directory:

job <- ss_job_attach("~/ss-runs/noise-vnl", cohort = cohort, cols = cols)
ss_job_status(job)
res <- ss_job_result(job)          # or ss_job_cancel(job) to stop it

7. Your own data

Register a dataset once and link it by name in later sessions, like a catalog variable. Registration checks and prepares the files and stores them, with a record of their metadata, in a registry directory (see ?ss_datasets); your original files are no longer needed by the link. A dataset is a table keyed by a Census geography (static, yearly or daily), GeoTIFF rasters (static, yearly, biweekly or daily, on one grid), or point, line or polygon features, whose variable is the distance to the nearest feature. This example needs no catalog data:

# Registering your own data once and linking it by name. The dataset here
# is three points (city parks) written as GeoJSON; the variable is the
# distance in meters from each location to the nearest point. It needs no
# catalog data, only the Python side (ss_install()):
#   source(system.file("examples", "registered-dataset.R", package = "spacescans"))
library(spacescans)

cohort <- data.frame(
  pid = c("P1", "P2", "P2", "P3"),
  startDate = as.Date(c("2015-01-01", "2014-03-01", "2016-07-01", "2017-01-01")),
  endDate = as.Date(c("2017-12-31", "2016-06-30", "2019-12-31", "2018-06-30")),
  longitude = c(-86.1581, -87.6298, -84.3880, -122.4194),
  latitude = c(39.7684, 41.8781, 33.7490, 37.7749)
)

# GeoJSON coordinates are longitude and latitude (WGS 84).
parks <- file.path(tempdir(), "example_parks.geojson")
writeLines('{"type": "FeatureCollection", "features": [
  {"type": "Feature", "properties": {"name": "A"}, "geometry": {"type": "Point", "coordinates": [-86.1700, 39.7800]}},
  {"type": "Feature", "properties": {"name": "B"}, "geometry": {"type": "Point", "coordinates": [-87.6200, 41.8800]}},
  {"type": "Feature", "properties": {"name": "C"}, "geometry": {"type": "Point", "coordinates": [-84.3700, 33.7600]}}
]}', parks)

# Prepared once and stored in the registry (see ?ss_datasets for where).
rec <- ss_register_dataset(parks, name = "example_parks", kind = "features", temporal = "static",
                           description = "Three example park locations.", overwrite = TRUE)
print(rec)
print(ss_datasets())

# Linked by name, like a catalog variable; no catalog variable is needed here.
data_dir <- tryCatch(ss_data_dir(), spacescans_error_config = function(e) tempdir())
res <- ss_link(cohort, character(), custom = "example_parks", data_dir = data_dir)
print(res)                        # pid and example_parks_dist_m, one row per participant

ss_remove_dataset("example_parks")

ss_datasets() lists what is registered, ss_dataset() shows one record, ss_update_dataset() changes its name, metadata or variable units, ss_remove_dataset() deletes one, and ss_link(custom = ) takes names, ids or records, alone or with catalog variables. A table names its Census geography in key, with the key column when it is not called GEOID10, GEOID or the like:

svi <- ss_register_dataset("~/data/svi_2020_tract.csv", name = "SVI 2020",
                           kind = "polygon_table", temporal = "static",
                           key = list(boundary = "tract", column = "FIPS"),
                           variables = data.frame(name = "RPL_THEMES", unit = "percentile",
                                                  description = "Overall SVI percentile"),
                           source = "CDC/ATSDR SVI 2020", version = "2020")
res <- ss_link(cohort, "ndi", custom = "SVI 2020", cols = cols)

# Rasters: one GeoTIFF per period; the period comes from the file name or `periods`.
pm <- ss_register_dataset(c("pm_2016a.tif", "pm_2016b.tif"), name = "PM2.5 biweekly",
                          kind = "raster", temporal = "biweekly",
                          periods = as.Date(c("2016-01-01", "2016-01-15")))

For a one-off dataset that you do not want to register, ss_custom_polygon() (a CSV keyed by a Census geography, static or yearly) and ss_custom_raster() (GeoTIFFs, one or one per year) build a definition that points at your files where they are:

own <- ss_custom_polygon("svi_tract.csv", boundary = "Tract", key_col = "FIPS",
                         value_cols = "RPL_THEMES", name = "CDC SVI")
res <- ss_link(cohort, "ndi", custom = own, cols = cols)

8. When something goes wrong

Every error inherits from spacescans_error and has a class for its kind, so a script can react to exactly what it expects:

res <- tryCatch(
  ss_link(cohort, "faqsd", cols = cols),
  spacescans_error_data_missing = function(e) {
    message(conditionMessage(e))    # includes the ss_fetch() call to run
    NULL
  }
)

See ?"spacescans-conditions" for the list. For one pipeline config of your own (the lower-level interface), see ss_run(), ss_templates() and ss_validate().