Skip to content

About

Status of the ICES R packages

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

ICES R packages status

Source for https://ices-tools-prod.github.io, a summary of the status of the R packages maintained by ICES, and of the web services they use. The site is built with Quarto.

Adding or removing packages

The packages shown are listed in packages.json. Each entry needs the package name and its GitHub repository:

{ "package": "icesVMS", "repo": "ices-tools-prod/icesVMS" }

An optional "notes" field is shown on the package page. Private repositories are left out, as the site is public. After editing the file on main, the site is rebuilt automatically.

Web services

Each web service (API) that is checked has a folder in apis/, named by its id (e.g. apis/sag/), containing:

  • config.json: the name, base url, documentation, the packages that call it, notes shown in a table on its page, and the endpoints to check. Each endpoint has a path with example query values, which should return a stable result (for example a published assessment), and the R functions that call it. An endpoint can also give:
    • "auth": true if it needs an ICES login. It is then called with an ICES token, taken from the ICES_TOKEN environment variable or, if that is not set, the token stored by icesConnect. Without a token the endpoint is shown as not checked, and recording a baseline keeps its previous one. In the GitHub Action the token is the ICES_TOKEN repository secret (see below);
    • "template": its path with parameters in braces, such as footprint/passive/{ecoregion}/{year}, for APIs with no OpenAPI description, so that the operation is named by it rather than by the example values.
  • baseline.json: the recorded structure of each endpoint's response, and the documented operations and parameters.

To add an API, copy a folder, edit config.json, and record its baseline (see below).

Every day, each endpoint is called and the structure of its response (the field names, their types, and the format of text values such as dates and numbers stored as text) is compared with the baseline. If the API has an OpenAPI (swagger) description, set as spec in config.json, the documented operations and their parameters are compared too. Changes are reported on the site: fields renamed (including changes of case, such as AssessmentKey to assessmentKey), added or removed, fields changing type, and changes in the HTTP status or content type.

The R code of the packages that call each API is read from GitHub (parsed, never run) to find the R function for each operation, and the functions that combine several operations. Operations that read data but have no R function are listed with a suggested function name.

When a change is expected, or for a new API, record a new baseline for it and commit it:

Rscript scripts/check_apis.R --update-baseline sag
git add apis/sag/baseline.json

Without an id, the baselines of all APIs are updated.

The ICES token

The ICES_TOKEN repository secret is a personal ICES token, which expires after a few weeks. The page of each API that uses it shows when it expires; once it has expired, the endpoints that need a login are reported as errors. To update it, log in at https://data.ices.dk/token, set the token in R as shown there, and run:

Rscript -e 'cat(icesConnect::ices_token())' | gh secret set ICES_TOKEN --repo ices-tools-prod/ices-tools-prod.github.io

Files

File Contents
packages.json the packages shown
apis/<id>/config.json a web service and its endpoints
apis/<id>/baseline.json the recorded structure of its responses and documentation
index.qmd the overview page
packages.qmd the table of all packages
apis.qmd the overview of the web services
_package.qmd template for the package pages
_api.qmd template for the web service pages
R/status.R R functions used by the pages
scripts/build_status.R collects the status of each package into data/status.json
scripts/check_apis.R checks the web services, into data/apis.json
scripts/api_helpers.R functions used by check_apis.R
scripts/make_pages.R creates a page per package (package/) and per web service (api/) from the templates
_quarto.yml site settings: navigation, theme, footer, and the scripts run before rendering
.github/workflows/build.yml collects the data when needed, renders and publishes the site

data/, package/ and api/ are created when the site is built, and are not committed.

How it works

A GitHub Action (.github/workflows/build.yml) runs every day at 05:00 UTC, on every push to main, and on demand from the Actions tab. It has two parts:

  • data: runs scripts/build_status.R and scripts/check_apis.R, and saves data/ as an artifact. This runs every day, on pushes that change packages.json, apis/ or those scripts (and scripts/api_helpers.R), and on runs from the Actions tab with "refresh data" ticked;
  • render: renders the pages with quarto render and publishes them, using the data just collected or, if it was not collected, the data saved by the latest run that collected it.

So a push that only changes the pages or how they show the data (the .qmd files, R/status.R, _quarto.yml) is published in a minute or two, without checking the web services again. The render runs in the rocker/r-ver container, which has R installed, and the R packages are cached between runs; the cache is renewed each month and whenever the workflow file changes. The status collected for each package is:

  • GitHub: version in DESCRIPTION on the default branch, maintainer, open issues and pull requests, last commit;
  • CRAN: published version, check results over all flavours, additional issues, deadlines, and whether the package has been archived (and why);
  • r-universe: build status and check results, from the ices-tools-prod and ices-tools-dev universes;
  • dependencies: the hard dependencies (Depends, Imports, LinkingTo), found recursively from CRAN and the ICES r-universes, excluding base R packages.

Building locally

Needs Quarto, the R packages jsonlite, curl, xml2, knitr, rmarkdown, htmltools and reactable, and the GitHub CLI, logged in.

quarto preview

The status data is only collected again if it is more than 6 hours old. To force it:

STATUS_REFRESH=true quarto render

To change how the pages look without collecting the data again, however old it is:

STATUS_REFRESH=false quarto preview

Status levels

  • Action needed: CRAN check errors, a CRAN deadline, the GitHub repository or DESCRIPTION file is missing, or a web service endpoint returns an error.
  • Needs attention: archived on CRAN, CRAN check warnings or additional issues, failing r-universe builds or checks, the GitHub version is older than the CRAN version, or a web service response has changed.
  • OK: none of the above. Unreleased changes and packages not on the r-universe are shown for information.

About

Status of the ICES R packages

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages