Run OceanBench offline

The compute nodes of many HPC clusters have no internet access. OceanBench can still evaluate a challenger there, using data downloaded beforehand on a node that has internet access.

Two steps

The two nodes must share a file system, for example a scratch or work directory.

  1. On a node with internet access, such as a login node, download the data the evaluation needs:

    oceanbench stage path/to/challenger.py --stage-dir /shared/oceanbench-stage

    This command only downloads data. It computes no score, so it is light enough for a login node.

  2. On the offline compute node, run the evaluation from the downloaded data:

    oceanbench evaluate path/to/challenger.py --stage-dir /shared/oceanbench-stage

    Step 2 needs the challenger file on a local disk. A challenger given as a URL is downloaded, and --all-challengers asks the GitHub API for the challenger list, so neither works offline.

The evaluation finds everything in /shared/oceanbench-stage and does not download anything. The directory must be writable from the compute node. For a challenger that uses one of the challenger datasets provided by OceanBench (oceanbench.datasets.challenger), step 2 also writes there the surface currents used by the Lagrangian scores. They take about 0.1 GB per forecast week at 1/4 degree. For example, one GLONET week took 4.8 GB after step 1 and 4.9 GB after step 2. A challenger file that opens its forecasts with plain xarray writes no such files.

To evaluate a region, give both commands the same region, with --region, --region-file or the OCEANBENCH_REGION environment variable.

What gets staged

oceanbench stage writes these datasets to the stage directory:

  • the reference datasets, GLORYS reanalysis and GLO12 analysis
  • the observations used by the Class IV scores
  • the mean dynamic topography used to compare sea level with observations
  • the OceanBench ocean mask
  • the challenger forecasts, when the challenger file uses one of the challenger datasets provided by OceanBench

A challenger file that opens its own forecasts must read them from a disk the offline node can see.

The native 1/12 degree references are read with the Copernicus Marine Toolbox. The node that runs step 1 therefore needs Copernicus Marine Service credentials, as described in the README. The offline node does not need them.

The stage directory belongs to one challenger

The staged data depends on the challenger: its forecast dates, its number of lead days and its depths. Use one stage directory per challenger, and give step 1 the same challenger file as step 2. A challenger built on top of one provided by OceanBench, for example a bias correction of GLONET, needs its own stage directory. Otherwise it reads the surface currents staged for the original challenger. If you change the forecast dates, the lead days, the depths or the region of a challenger, run step 1 again so that the missing data gets downloaded. Data already in the stage directory is not downloaded again.

Keep the stage directory

A folder given with --stage-dir or OCEANBENCH_STAGE_DIR is kept. You can evaluate the same challenger again without downloading everything again. OceanBench only deletes its default temporary folder, used when no folder is given. It deletes that folder after a successful oceanbench evaluate, unless --keep-stage is given.

A job killed while it writes to the stage directory can leave a *.lock directory behind, and the next run waits for it. Delete that *.lock directory before running again.

Environment variables

The same settings can be given as environment variables, for example in a batch script:

  • OCEANBENCH_STAGE_DIR=/shared/oceanbench-stage is the same as --stage-dir. On its own, it stages everything.
  • OCEANBENCH_STAGE selects only some datasets for oceanbench evaluate, like --stage. Several values can be separated by commas, for example references,observations. oceanbench stage always stages everything.
  • OCEANBENCH_STAGE_MAX_WORKERS is the same as --stage-max-workers, the number of parallel downloads.
  • OCEANBENCH_REMOTE_RETRIES is the same as --remote-retries, the number of attempts for each remote read.
  • OCEANBENCH_REGION is the same as --region.

Command line options take precedence over environment variables. --keep-stage has no environment variable and must be passed on the command line.