The USGS/EROS Machine-to-Machine (M2M) API provides programmatic access to the same catalogue that Earth Explorer serves, covering several hundred datasets. This vignette walks through the full path from logging in to having files on disk.
How the package is organised
Getting data out of the M2M API takes several steps, and they have to happen in a particular order: you cannot list a scene’s products before you have searched for the scene, and you cannot download anything before you have requested it.
Rather than leaving that order for you to remember, the package models it as a chain of objects. Each step returns an object whose methods are the steps that are valid next:
M2MSession → M2MDataset → M2MSceneSearch → M2MDownloadOptions → M2MDownloadQueueTwo practical consequences: pressing Tab after
$ shows only what you can legitimately do from where you
are, and every object prints a Next: line reminding you
where to go.
Authenticating
Authentication needs your ERS username and an M2M application token. The token is not your Earth Explorer password; generate one from your profile at https://ers.cr.usgs.gov/. Access to the M2M API itself must be requested and granted before any of this will work.
Keeping credentials out of your scripts is worth the small effort —
store them in ~/.Renviron via
usethis::edit_r_environ():
M2M_USERNAME=your_username
M2M_TOKEN=your_application_tokenm2m_session() reads those two variables by default:
sess <- m2m_session()
sess
#> <M2MSession>
#> User: your_username
#> Status: connected
#> Next: $dataset(name) | $find_datasets(pattern)Sessions expire after a period of inactivity. Call
sess$logout() when you are finished to release it
immediately.
Finding a dataset
Datasets are identified by an alias such as
landsat_ot_c2_l2. If you do not know the alias, search the
catalogue for a name pattern — wildcards are applied automatically:
sess$find_datasets("landsat_ot")
#> # A tibble: 2 × 21
#> datasetAlias collectionName sceneCount ...
#> <chr> <chr> <int>
#> 1 landsat_ot_c2_l1 Landsat 8-9 OLI/TIRS C2 L1 4791194
#> 2 landsat_ot_c2_l2 Landsat 8-9 OLI/TIRS C2 L2 4132193Once you know which one you want, pick it up by alias:
ds <- sess$dataset("landsat_ot_c2_l2")
ds
#> <M2MDataset>
#> Alias: landsat_ot_c2_l2
#> Name: Landsat 8-9 OLI/TIRS C2 L2
#> Scenes: 4,132,193
#> Next: $search(...) | $filters()The full metadata record is in ds$info, a one-row
tibble.
Searching for scenes
$search() takes any combination of a spatial, temporal,
cloud cover, and metadata filter. The filter_*() helpers
build them:
found <- ds$search(
spatial = filter_spatial(ll_lon = -120, ll_lat = 40, ur_lon = -119.5, ur_lat = 40.5),
temporal = filter_temporal("2020-07-01", "2020-07-31"),
cloud = filter_cloud(min = 0, max = 50)
)
found
#> <M2MSceneSearch>
#> Dataset: landsat_ot_c2_l2
#> Scenes: 3 of 3 total hits
#> Next: $products() | $filter(...)Results are in the $scenes tibble. Searches are paged
through exhaustively by default; pass max_results to stop
early. $total_hits always reports what the API matched
overall, so comparing it against nrow(found$scenes) tells
you whether you are looking at a complete result set:
found$scenes
found$total_hits$filter() narrows the scenes locally, using
dplyr::filter() semantics against the $scenes
tibble. No further API call is made:
recent <- found$filter(cloudCover < 10)
recent$scenesDataset-specific filters
Beyond the common filters, each dataset exposes its own searchable
metadata fields. $filters() lists them, one row per
field:
ds$filters()
#> # A tibble: 12 × 7
#> id fieldLabel valueList ...
#> <chr> <chr> <list>
#> 1 5e83d14f567d0086 Landsat Product Identifier L2 <NULL>
#> 4 5e83d14fb9436d88 WRS Path <NULL>
#> 5 5e83d14ff1eda1b8 WRS Row <NULL>
#> 6 61af9273566bb9a8 Satellite <list [3]>The id column gives the filterId that the
filter_metadata_*() builders take. For fields of type
Select, valueList holds the permitted
values.
Searching a specific Landsat path/row, for instance:
ds$search(
temporal = filter_temporal("2020-07-01", "2020-07-31"),
metadata = filter_metadata_and(
filter_metadata_value("5e83d14fb9436d88", "043"), # WRS Path
filter_metadata_value("5e83d14ff1eda1b8", "032") # WRS Row
)
)filter_metadata_between() covers ranges, and
filter_metadata_or() combines alternatives.
Choosing what to download
$products() asks the API what is actually downloadable
for the scenes you found:
opts <- found$products()
opts
#> <M2MDownloadOptions>
#> Scenes: 12
#> Files: 66 selected
#> Next: $request(label) | $select_bands(...) | $bands()Behind the scenes this registers a private scene list on the M2M server, which the API requires before it will list download options. The package owns that plumbing; the search interface remains focused on scenes and products.
$bands() returns one row per downloadable file:
opts$bands()
#> # A tibble: 66 × 11
#> entityId displayId bulkAvailable filesize
#> <chr> <chr> <lgl> <int>
#> 1 L2SR_LC08_L2SP_043032_20200717_..._B4_TIF LC08_..._SR_B4 TRUE 62914560
#> 2 L2SR_LC08_L2SP_043032_20200717_..._B5_TIF LC08_..._SR_B5 TRUE 62914560A Landsat scene has dozens of associated files, so you will usually
want a subset. $select_bands() matches literal substrings
of displayId, which is how "B4" picks out
..._SR_B4, and $filter() accepts arbitrary
dplyr expressions. Both return a new object, leaving the
original selection intact:
Whole products versus individual files
The API offers downloads at two granularities, and
$bands() only shows one of them. Alongside the individual
band files, each scene has whole-product entries: a Level-1 Product
Bundle, which is a single .tar holding the entire scene,
and full-resolution browse images. $scene_products() lists
these:
opts$scene_products()
#> # A tibble: 10 × 12
#> productName available id
#> <chr> <lgl> <chr>
#> 1 Landsat Collection 2 Level-1 Product Bundle TRUE 5e83d0a0f94d7d8d
#> 2 Full-Resolution Browse (Natural Color) GeoTIFF TRUE 5e9eb4f5ebb15d46
#> 3 Full-Resolution Browse (Thermal) GeoTIFF TRUE 5e9eb5318675b02d
#> 8 Landsat Collection 2 Level-1 Product Bundle FALSE 63231219fdd8c4e5Browse images have no constituent files, so they never appear in
$bands() at all. Use $select_products() to
queue whole products instead of individual files — for one
.tar per scene rather than 19 separate downloads:
bundles <- opts$
select_products("Landsat Collection 2 Level-1 Product Bundle")$
filter(available)
bundles$request(label = "bundles_2020")That name is matched exactly, against
productName or productCode, so it is a value
copied straight out of the listing above. Names are not standardised
across the catalogue — the bundle is “Landsat Collection 2 Level-1
Product Bundle” here and “Standard Format” for corona2 —
and they contain characters like parentheses that pattern matching would
misread, so read them off $scene_products() rather than
guessing. A value matching nothing warns and lists what was
available.
For looser selection, chain $filter() with any
expression you like:
opts$select_products()$filter(grepl("Browse", productName))Note the available filter too: the API lists superseded
products with available = FALSE, and requesting one will
fail. As with $select_bands(), neither selector filters on
availability for you.
$selected() always shows exactly what
$request() will queue, whichever granularity you chose.
Requesting and retrieving
$request() places the order. The label is how you find
the order again later, so give it something meaningful:
queue <- selected$request(label = "ndvi_july_2020")
queue
#> <M2MDownloadQueue>
#> Label: ndvi_july_2020
#> Available: 6 file(s) ready
#> Preparing: 0
#> Next: $retrieve(out_dir)A file is ready once it has a URL. Some come straight back that way;
others have to be staged first. The API’s internal available and
requested buckets do not map cleanly onto that — proxied downloads can
be in the requested bucket but already carry a working URL — so use
$ready() and $pending().
$is_ready() tells you whether anything is still staging,
and $refresh() re-polls:
while (!queue$is_ready()) {
Sys.sleep(30)
queue$refresh()
}$refresh() is the one method in the package that
modifies its object in place, because it reflects live server-side
state. Everything else returns a new object.
Finally, write the files to disk. The directory is created if it does not exist:
results <- queue$retrieve("data/landsat")
results
#> # A tibble: 6 × 4
#> entityId url path status
#> <chr> <chr> <chr> <chr>
#> 1 L2SR_LC08_..._SR_B4_TIF https://landsatlook... data/landsat/L2SR... downloadedCheck results$status for any "failed" rows
rather than assuming everything arrived.
Resuming a large order
Orders live on the server, so a download does not have to finish in one sitting. Reconnect by label from a fresh session:
sess <- m2m_session()
queue <- sess$download_queue("ndvi_july_2020")
queue$refresh()
queue$retrieve("data/landsat")If you no longer remember the label,
sess$download_labels() lists your orders one row each:
sess$download_labels()
#> # A tibble: 1 × 5
#> label totalComplete downloadCount downloadSize dateEntered
#> <chr> <int> <int> <chr> <dbl>
#> 1 ndvi_july_2020 0 6 549580314 1787113928127sess$downloads() lists every individual download across
all labels, and queue$summary() breaks one order down by
dataset. queue$cancel() abandons an order you no longer
want.
Orders whose scenes are staged rather than queued need
queue$prepare() to start them processing — it is the one
queue method besides $cancel() that changes server-side
state:
queue$prepare()
queue$refresh()Related scenes
Some datasets define a relationship between scenes — a mosaic and its
source frames, for instance. $related_scenes() follows
it:
ds <- sess$dataset("aerial_combin")
related <- ds$related_scenes("AR1MSH06B001001", max_results = 5)
related$scenesMost datasets define no such relationship and raise an
m2m_error with code DATASET_ERROR. Because
related scenes usually belong to a different dataset, the
returned search is bound to that dataset, so $products() on
the result acts on the right one.