Ordering Sentinel-2 products from Long Term Archive with sen2r

R on Luigi Ranghetti Website

2 years ago

[This article was first published on R on Luigi Ranghetti Website, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Until August 2019, all Sentinel-2 satellite data could be directly downloaded from the ESA Data Hub, both through the interactive Open Hub or using an API interface.

Recently this policy was changed: typically, only the most recent products are available for direct download, while the oldest ones (level 2A archives older than 18 months and level 1C older than one year) are stored in the so called Long Term Archive, and must be ordered by the user; then, they are made available for download after a while (no messages are sent to the user). For further details, see the announcement of activation of LTA access for Sentinel-2 and 3.

R users can exploit a functionality recent implemented in the package sen2r to deal with products no more available for direct download.

Overview and installation

sen2r is a package devoted to download and preprocess Sentinel-2 satellite imagery via an accessible and easy to install interface, which can also be used to manage massive processing operations and to schedule automatic processing chains. For an overview of the packages functionalities, see this previous post.

sen2r was recently released on CRAN, but the current CRAN release (1.1.0) does not yet support ordering products from the Long Term Archive (LTA); to exploit this functionality, it is necessary to use the GitHub release (version 1.2.0 or higher), which can be installed using the package remotes:

remotes::install_github("ranghetti/sen2r")

(refer to this page for a detailed installation guide, including the installation of system dependences).

After installing the package, the following function should be used to save the SciHub credentials:

library(sen2r)
write_scihub_login("my_user", "my_password")

Automatic orders

sen2r can be used both in online and offline mode: in the first case, Sentinel-2 source archives (in SAFE format) are searched and downloaded from ESA SciHub, while in the second case, only archives already present on the user machine are used. This post refers to the online usage of the package (nothing changes using it in offline mode).

Starting from version 1.2.0, SAFE archives which are not available for direct download are automatically ordered: this means that, in the case some archives – required to produce the users’ outputs – are stored in the LTA:

they are ordered;
the processing chain launched by the user continues skipping these archives (and so without producing the consequent output rasters);
the user can check the order status with a specific command.

Let’s see some examples of that.

Note: in the following examples, we will search / download an area of interest located in a random Sentinel-2 tile of Kazakhstan, in a time window of 20 days. Randomness is necessary tyo catch products not available for download (otherwise, archives overlapping a fixed area of interest would be ordered the first time I ran examples, so making them available and compromising the subsequent executions). This expedient does not ensure reproducibility, since the randomly selected products could already be available for different reasons.

Using the function `sen2r()`

sen2r() is the main function of the package, which can be used to manage a whole processing chain. In the following example sen2r() will be used to launch a processing chain to generate RGB colour images from level 1C (top of atmosphere reflectances) data. First of all, the boundaries of Kazakhstan are downloaded and used to select the Sentinel-2 tiles intersecting this country. Then, a specific area of interest (the centroid of a random tile, with a buffer of 5 km) is created.

library(sf); library(magrittr)
download.file(
  "https://biogeo.ucdavis.edu/data/gadm3.6/Rsf/gadm36_KAZ_0_sf.rds",
  kaz_sf_path <- file.path(tempdir(), "gadm36_KAZ_0_sf.rds")
)
kaz_sf <- readRDS(kaz_sf_path)

s2tiles_kaz <- tiles_intersects(kaz_sf, all = TRUE, out_format = "sf")

In this example, tiles_intersects() is a sen2r function which allows obtaining the Sentinel-2 tiles which cover a specific area of interest. Then, the processing is launched:

safe_folder <- tempfile(pattern = "safe_")
out_folder_1 <- tempfile(pattern = "Example1_")
sel_tile_1 <- s2tiles_kaz[sample(nrow(s2tiles_kaz),1),] %>%
  sf::st_transform(3857)
sel_tile_1$tile_id
## [1] "40TET"
sel_extent_1 <- sf::st_centroid(sel_tile_1) %>% st_buffer(5e3)
out1 <- sen2r(
  gui = FALSE,
  extent = sel_extent_1,
  timewindow = c("2018-02-21", "2018-03-02"),
  list_rgb = "RGB432T",
  path_l1c = safe_folder,
  path_l2a = safe_folder,
  path_out = out_folder_1,
  log = log_path_1 <- tempfile()
)

This is the log of the function (by default it is returned at standard output, while in this example it was redirected to a temporary file by setting the argument log):

## [2019-10-24 15:27:02] Starting sen2r execution.
## [2019-10-24 15:27:02] Searching for available SAFE products on SciHub...
## [2019-10-24 15:27:09] Ordering 1 Sentinel-2 images stored in the Long Term Archive...
## [2019-10-24 15:27:09] 1 of 1 Sentinel-2 images were correctly ordered. You can check at a later time if the ordered products were made available using the command:
## 
## safe_is_online("/home/lranghetti/.sen2r/lta_orders/lta_20191024_152709.json")
## 
## [2019-10-24 15:27:09] Computing output names...
## [2019-10-24 15:27:10] Starting to download the required level-2A SAFE products.
## [2019-10-24 15:27:10] Download of level-2A SAFE products terminated.
## [2019-10-24 15:27:10] Starting to download the required level-1C SAFE products.
## [2019-10-24 15:27:10] Check if products are available for download...
## [2019-10-24 15:27:11] 1 Sentinel-2 images are already available and will not be ordered.
## [2019-10-24 15:27:11] Downloading Sentinel-2 image 1 of 1 (S2B_MSIL1C_20180301T070819_N0206_R106_T40TET_20180301T105657.SAFE)...
## [2019-10-24 15:28:45] Download of level-1C SAFE products terminated.
## [2019-10-24 15:28:45] Updating output names...
## [2019-10-24 15:28:45] Starting to translate SAFE products in custom format.
## GDAL version in use: 2.2.3
## Using UTM zone 40.
## 1 output files were correctly created.
## [2019-10-24 15:28:45] Starting to merge tiles by orbit.
## Using projection "+proj=utm +zone=40 +datum=WGS84 +units=m +no_defs".
## [2019-10-24 15:28:46] Starting to edit geometry (clip, reproject, rescale).
## [2019-10-24 15:28:46] Producing required RGB images.
## 1 output RGB files were correctly created.
## [2019-10-24 15:28:50] Generating thumbnails.
## 1 output files were correctly created.
## [2019-10-24 15:28:51] Execution of sen2r session terminated.

The required archive, not available for direct download, were ordered. After doing that, function continued processing the available archives.

At the end of the processing, a warning could appear in case the user exceeded the order quota:

## <n> of <N> Sentinel-2 images were not correctly ordered because user '<username>' offline products retrieval quota exceeded. Please retry later, otherwise use different SciHub credentials (see ?write_scihub_login or set a specific value for argument "apihub").

The function cited in the log can be used to check if the order was processed:

## safe_is_online("/home/lranghetti/.sen2r/lta_orders/lta_20191024_152709.json")
## S2A_MSIL1C_20180224T070851_N0206_R106_T40TET_20180224T110454.SAFE 
##                                                             FALSE

(the JSON file, automatically created by sen2r(), contains the URLs of the ordered products).

Whenever this function will be returning TRUE for all the listed archives, the previous sen2r() execution can be re-launched to complete the processing.

In the case the user does not want to order missing products, it is sufficient to set the sen2r() argument order_lta to FALSE. Launching the function with the Graphical User Interface, this setting can be set in the first sheet (see the screenshot below).

Using specific functions

Other package functions can be used to perform specific steps of a processing chain:

s2_list() allows searching the SAFE archives matching the required parameters. By default, all archives (downloadable / stored in LTA) are returned; setting the new argument availability to "online" or "lta" allows returning only available archives or LTA products, respectively. Setting availability = "check" allows distinguish which returned products are available for download and which are not:

out_folder_2 <- tempfile(pattern = "Example2_")
sel_tile_2 <- s2tiles_kaz[sample(nrow(s2tiles_kaz),1),]
sel_tile_2$tile_id
## [1] "42UXF"
out_list_2 <- s2_list(
  tile = sel_tile_2$tile_id, 
  time_interval = c("2018-02-21", "2018-03-02"),
  availability = "check"
)
# Show available products
out_list_2[attr(out_list_2, "online")]
##                                      S2A_MSIL1C_20180301T061801_N0206_R034_T42UXF_20180301T082121.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('ef9d324a-1172-42f5-8396-b120ab4dab89')/$value" 
##                                      S2B_MSIL1C_20180302T063759_N0206_R120_T42UXF_20180302T102521.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('9efefd9f-563c-4ab4-bdb1-e7178688c551')/$value"
# Show products stored in LTA
out_list_2[attr(out_list_2, "lta")]
##                                      S2A_MSIL1C_20180222T062851_N0206_R077_T42UXF_20180222T101014.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('e6f94a43-804c-4fa7-a216-8ffc6620b8ed')/$value" 
##                                      S2B_MSIL1C_20180224T061829_N0206_R034_T42UXF_20180224T101548.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('7766d2f8-da66-427f-8e19-b134c4c482f3')/$value" 
##                                      S2A_MSIL1C_20180225T063831_N0206_R120_T42UXF_20180225T103712.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('bff08819-ce0e-44b5-8327-4e379fc7542c')/$value" 
##                                      S2B_MSIL1C_20180227T062809_N0206_R077_T42UXF_20180227T083242.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('d44b1929-5b77-47a1-865c-7515dcab9960')/$value"

s2_download() can be used to download specific SAFE products. By default, products stored in LTA are automatically ordered (unless argument order_lta is set to FALSE by the user):
```
s2_download(out_list_2, order_lta = FALSE)
```
This function can also be used passing it the path of a JSON file created by sen2r() (see above) or by s2_order() (see below) containing the URLs of the archives to be downloaded.

safe_is_online() function, which was already shown, can also be used with a list of required archives:

safe_is_online(out_list_2)
## S2A_MSIL1C_20180222T062851_N0206_R077_T42UXF_20180222T101014.SAFE 
##                                                             FALSE 
## S2B_MSIL1C_20180224T061829_N0206_R034_T42UXF_20180224T101548.SAFE 
##                                                             FALSE 
## S2A_MSIL1C_20180225T063831_N0206_R120_T42UXF_20180225T103712.SAFE 
##                                                             FALSE 
## S2B_MSIL1C_20180227T062809_N0206_R077_T42UXF_20180227T083242.SAFE 
##                                                             FALSE 
## S2A_MSIL1C_20180301T061801_N0206_R034_T42UXF_20180301T082121.SAFE 
##                                                              TRUE 
## S2B_MSIL1C_20180302T063759_N0206_R120_T42UXF_20180302T102521.SAFE 
##                                                              TRUE

Manual orders

The new function s2_order() can be used to manually order SAFE archives stored on LTA. Similarly to s2_download() and safe_is_online(), s2_order() accepts a list of SAFE URLs to be downloaded, both as R vector of as path of a JSON file containing them:

ordered_list <- s2_order(out_list_2)
## [2019-10-24 15:29:19] Check if products are already available for download...
## [2019-10-24 15:29:22] 2 Sentinel-2 images are already available and will not be ordered.
## [2019-10-24 15:29:22] Ordering 4 Sentinel-2 images stored in the Long Term Archive...
## [2019-10-24 15:29:38] 4 of 4 Sentinel-2 images were correctly ordered. You can check at a later time if the ordered products were made available using the command:
## 
## safe_is_online("/home/lranghetti/.sen2r/lta_orders/lta_20191024_152938.json")
ordered_list
##                                      S2A_MSIL1C_20180222T062851_N0206_R077_T42UXF_20180222T101014.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('e6f94a43-804c-4fa7-a216-8ffc6620b8ed')/$value" 
##                                      S2B_MSIL1C_20180224T061829_N0206_R034_T42UXF_20180224T101548.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('7766d2f8-da66-427f-8e19-b134c4c482f3')/$value" 
##                                      S2A_MSIL1C_20180225T063831_N0206_R120_T42UXF_20180225T103712.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('bff08819-ce0e-44b5-8327-4e379fc7542c')/$value" 
##                                      S2B_MSIL1C_20180227T062809_N0206_R077_T42UXF_20180227T083242.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('d44b1929-5b77-47a1-865c-7515dcab9960')/$value" 
## attr(,"available")
##                                      S2A_MSIL1C_20180301T061801_N0206_R034_T42UXF_20180301T082121.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('ef9d324a-1172-42f5-8396-b120ab4dab89')/$value" 
##                                      S2B_MSIL1C_20180302T063759_N0206_R120_T42UXF_20180302T102521.SAFE 
## "https://scihub.copernicus.eu/apihub/odata/v1/Products('9efefd9f-563c-4ab4-bdb1-e7178688c551')/$value" 
## attr(,"notordered")
## named character(0)
## attr(,"path")
## [1] "/home/lranghetti/.sen2r/lta_orders/lta_20191024_152938.json"

The function returns a vector of ordered products. Some attributes are eventually added:

"available" contains products not ordered because already available for download;
"notordered" contains products whose orders failed (commonly because the user exceeded his quota);
"path" contains the path of the saved JSON file containing the URLs of the ordered products, which can be used with functions safe_is_online() and s2_download() to check if the order was processed and then to download products (the creation of this file can be skipped setting export_prodlist = FALSE).

Conclusions

Starting from version 1.2.0, sen2r is able to manage Sentinel-2 products stored in the Long Term Archive (LTA) and so not directly downloadable. The default behaviour of package functions is to order unavailable products, processing only available archives and providing the way to check the order status through a single line of code. Once the order was processed, the user can relaunch the same code to complete the processing. New functions can be exploited to perform specific steps: safe_is_online() can be used to check if SAFE archives are / were made available, and s2_order() to manually order them. The implementation of additional features related to LTA is planned for the future (e.g. an additional HTML report with the list of used / ordered products).

These features were implemented recently, so users could encounter some issues exploiting them. With the exception of issues not depending on this package (e.g. errors due to invalid SciHub credentials, or if user quota exceeded), users are encouraged to report them on GitHub.

Credits

sen2r is developed by Luigi Ranghetti and Lorenzo Busetto (IREA-CNR), and it is released under the GNU GPL-3 license.

Using sen2r for production (including scientific products) requires to cite it (use this entry).

To leave a comment for the author, please follow the link and comment on their blog: R on Luigi Ranghetti Website.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.