Source for https://ices-tools-prod.github.io, a summary of the status of the R packages maintained by ICES, and of the web services they use. The site is built with Quarto.
The packages shown are listed in packages.json. Each entry
needs the package name and its GitHub repository:
{ "package": "icesVMS", "repo": "ices-tools-prod/icesVMS" }An optional "notes" field is shown on the package page. Private
repositories are left out, as the site is public. After editing the file on
main, the site is rebuilt automatically.
Each web service (API) that is checked has a folder in apis/, named
by its id (e.g. apis/sag/), containing:
config.json: the name, base url, documentation, the packages that call it, notes shown in a table on its page, and the endpoints to check. Each endpoint has a path with example query values, which should return a stable result (for example a published assessment), and the R functions that call it. An endpoint can also give:"auth": trueif it needs an ICES login. It is then called with an ICES token, taken from theICES_TOKENenvironment variable or, if that is not set, the token stored by icesConnect. Without a token the endpoint is shown as not checked, and recording a baseline keeps its previous one. In the GitHub Action the token is theICES_TOKENrepository secret (see below);"template": its path with parameters in braces, such asfootprint/passive/{ecoregion}/{year}, for APIs with no OpenAPI description, so that the operation is named by it rather than by the example values.
baseline.json: the recorded structure of each endpoint's response, and the documented operations and parameters.
To add an API, copy a folder, edit config.json, and record its baseline
(see below).
Every day, each endpoint is called and the structure of its response (the
field names, their types, and the format of text values such as dates and
numbers stored as text) is compared with the baseline. If the API has an
OpenAPI (swagger) description, set as spec in config.json, the
documented operations and their parameters are compared too.
Changes are reported on the site: fields renamed (including changes of case,
such as AssessmentKey to assessmentKey), added or removed, fields
changing type, and changes in the HTTP status or content type.
The R code of the packages that call each API is read from GitHub (parsed, never run) to find the R function for each operation, and the functions that combine several operations. Operations that read data but have no R function are listed with a suggested function name.
When a change is expected, or for a new API, record a new baseline for it and commit it:
Rscript scripts/check_apis.R --update-baseline sag
git add apis/sag/baseline.jsonWithout an id, the baselines of all APIs are updated.
The ICES_TOKEN repository secret is a personal ICES token, which expires
after a few weeks. The page of each API that uses it shows when it expires;
once it has expired, the endpoints that need a login are reported as errors.
To update it, log in at https://data.ices.dk/token, set the token in R as
shown there, and run:
Rscript -e 'cat(icesConnect::ices_token())' | gh secret set ICES_TOKEN --repo ices-tools-prod/ices-tools-prod.github.io| File | Contents |
|---|---|
packages.json |
the packages shown |
apis/<id>/config.json |
a web service and its endpoints |
apis/<id>/baseline.json |
the recorded structure of its responses and documentation |
index.qmd |
the overview page |
packages.qmd |
the table of all packages |
apis.qmd |
the overview of the web services |
_package.qmd |
template for the package pages |
_api.qmd |
template for the web service pages |
R/status.R |
R functions used by the pages |
scripts/build_status.R |
collects the status of each package into data/status.json |
scripts/check_apis.R |
checks the web services, into data/apis.json |
scripts/api_helpers.R |
functions used by check_apis.R |
scripts/make_pages.R |
creates a page per package (package/) and per web service (api/) from the templates |
_quarto.yml |
site settings: navigation, theme, footer, and the scripts run before rendering |
.github/workflows/build.yml |
collects the data when needed, renders and publishes the site |
data/, package/ and api/ are created when the site is built, and are
not committed.
A GitHub Action (.github/workflows/build.yml) runs every day at 05:00
UTC, on every push to main, and on demand from the Actions tab. It has
two parts:
- data: runs
scripts/build_status.Randscripts/check_apis.R, and savesdata/as an artifact. This runs every day, on pushes that changepackages.json,apis/or those scripts (andscripts/api_helpers.R), and on runs from the Actions tab with "refresh data" ticked; - render: renders the pages with
quarto renderand publishes them, using the data just collected or, if it was not collected, the data saved by the latest run that collected it.
So a push that only changes the pages or how they show the data (the
.qmd files, R/status.R, _quarto.yml) is published in a minute or two,
without checking the web services again. The render runs in the
rocker/r-ver container, which has R
installed, and the R packages are cached between runs; the cache is renewed
each month and whenever the workflow file changes. The status collected for each
package is:
- GitHub: version in
DESCRIPTIONon the default branch, maintainer, open issues and pull requests, last commit; - CRAN: published version, check results over all flavours, additional issues, deadlines, and whether the package has been archived (and why);
- r-universe: build status and check results, from the
ices-tools-prodandices-tools-devuniverses; - dependencies: the hard dependencies (Depends, Imports, LinkingTo), found recursively from CRAN and the ICES r-universes, excluding base R packages.
Needs Quarto, the R packages jsonlite, curl, xml2, knitr, rmarkdown,
htmltools and reactable, and the GitHub CLI, logged in.
quarto previewThe status data is only collected again if it is more than 6 hours old. To force it:
STATUS_REFRESH=true quarto renderTo change how the pages look without collecting the data again, however old it is:
STATUS_REFRESH=false quarto preview- Action needed: CRAN check errors, a CRAN deadline, the GitHub
repository or
DESCRIPTIONfile is missing, or a web service endpoint returns an error. - Needs attention: archived on CRAN, CRAN check warnings or additional issues, failing r-universe builds or checks, the GitHub version is older than the CRAN version, or a web service response has changed.
- OK: none of the above. Unreleased changes and packages not on the r-universe are shown for information.