FAQ

Why OceanBench

1. What is OceanBench?

OceanBench is an open-source benchmark to evaluate ocean forecasting systems against reference datasets (GLORYS reanalysis, GLO12 analysis) and in situ and satellite observations. It is developed by Mercator Ocean International with academic partners and published at NeurIPS 2025.

2. What does it add compared to evaluating my model on my own?

Every system is scored with the same references, period, masks and metric definitions, so scores are comparable across physics-based, machine-learning and hybrid systems. Scores are reproducible, and can be downloaded as CSV, Parquet or Zarr.

3. Who is it for?

• System developers who want an independent, standard evaluation.
• Users choosing a forecast for a given need (currents, sea level, temperature, salinity).
• Researchers who need reference training and evaluation datasets.

4. Does one score say which system is best?

No. Users have different priorities, so OceanBench reports many metrics per variable, depth and lead day. They capture key aspects of forecast quality, not the complete picture.

How it works

5. How is a system evaluated?

OceanBench executes a Python notebook that opens the challenger dataset (the forecasts of the system) as an xarray.Dataset and runs every evaluation method on it. The executed notebook is the evaluation report, and its content fills the score table.

6. What forecasts must a challenger provide?

All 10-day forecasts starting on the 52 Wednesdays of 2024. Resolutions of 1/12°, 1/4° and 1° are supported.

7. Which regions are covered?

The global ocean by default, and official regions such as IBI (--region ibi). Custom regions are defined with a bounding box in a JSON file (--region-file).

8. What is the 1 degree track?

A track for systems running at 1° resolution. It uses dedicated 1° GLORYS and GLO12 references, and interpolates higher-resolution challengers to a regular 1° grid for comparison. See the documentation.

9. How often are scores updated?

All official challengers are re-evaluated with each new OceanBench version, and the scores of every version stay available on the Scores page. They can be exported as CSV, Parquet or Zarr with the Export buttons, opened in the EDITO Data Explorer, or fetched through the Scores API.

The science

10. Which references are used?

• GLORYS reanalysis (2024)
• GLO12 analysis (2024)
• Argo profiles and surface drifters (in situ) and along-track altimetry (satellite)

11. Which metrics are computed?

Against each reference:

• RMSD of sea surface height, temperature, salinity and currents at 6 depths (surface, 50, 100, 200, 300 and 500 m).
• RMSD of mixed layer depth.
• RMSD of geostrophic currents.
• Deviation in kilometers of Lagrangian trajectories.

Against observations: Class IV RMSD per variable, depth bin and lead day.

Definitions are in the documentation.

12. How is the RMSD computed?

The mean of squared deviations is weighted by cos(latitude) so each cell counts in proportion to its area. Only ocean cells where both challenger and reference have a value are scored. Daily RMSD is then averaged over forecast initialization days.

13. What is Class IV?

The IV-TT CLASS-4 framework compares each forecast with observations at the observation time, position and depth. It is applied to the same observations for every challenger, whatever its grid.

14. Why compare to reanalysis, analysis and observations?

Each answers a different question:

• Observations: is the forecast close to what was measured?
• Reanalysis: does it reproduce the best available reconstruction of the ocean state?
• Analysis: does it match near-real time analysis?

Process-oriented diagnostics (mixed layer depth, geostrophic currents, Lagrangian trajectories) check that the physics is consistent, beyond fitting fields.

For challenger providers

15. How do I submit my system?

Open an issue on the GitHub repository with one of:

1. Access to the system output (Zarr, NetCDF).
2. Access to the system code or container with run instructions, so that the maintainers can reproduce the output.

Optionally, attach the executed notebook of an interactive or programmatic evaluation.

Add the organization, reference paper, method (physics-based, ML-based, hybrid), type (deterministic, ensemble), initial conditions and approximate resolution.

16. What format must my forecasts have?

An xarray.Dataset following CF conventions with:

• Dimensions: latitude, longitude, depth, lead_day_index (0 to 9), first_day_datetime.
• Variables: sea surface height above geoid, sea water potential temperature, salinity, and eastward and northward sea water velocity.

Names must follow CF standard names, or carry a standard_name attribute. Open it with explicit dask chunks. See the documentation.

17. Which data can I use to train and initialize my system?

import oceanbench

oceanbench.datasets.training.glorys_reanalysis()
oceanbench.datasets.training.glorys_reanalysis_1_degree()
oceanbench.datasets.input.glo12_nowcasts()
oceanbench.datasets.input.ifs_forcings()

GLORYS reanalysis is for training, GLO12 nowcasts and IFS forcings are the inputs for initial conditions and atmospheric forcing. The GLORYS training datasets cover 1993 to 2019 (included). Exclude the years covered by the benchmark, from any training data.

Run it yourself

18. How do I install OceanBench and what do I need?

pip install oceanbench

Evaluating 1/12° systems requires a Copernicus Marine Service account. For non-interactive runs, log in beforehand as described in the Copernicus Marine Toolbox documentation.

19. How do I evaluate a model end-to-end from the command line?

Write a Python file that defines a challenger_dataset variable, for example this sample:

from datetime import datetime
import xarray

challenger_dataset: xarray.Dataset = xarray.open_mfdataset(
    ["https://s3.waw3-1.cloudferro.com/oceanbench-bucket/public/ml-forecast-outputs/glonet/20240103.zarr"],
    engine="zarr",
    preprocess=lambda dataset: dataset.rename({"time": "lead_day_index"}).assign({"lead_day_index": range(10)}),
    combine="nested",
    concat_dim="first_day_datetime",
    parallel=True,
).assign({"first_day_datetime": [datetime.fromisoformat("2024-01-03")]})

Then run:

oceanbench evaluate challenger.py
oceanbench evaluate challenger.py --region ibi

The executed report is written as challenger.global.report.ipynb (or challenger.ibi.report.ipynb).

20. How do I evaluate from Python, or compute a single metric?

import oceanbench

oceanbench.evaluate_challenger("challenger.py")

import oceanbench
from challenger import challenger_dataset

oceanbench.metrics.rmsd_of_variables_compared_to_glorys_reanalysis(challenger_dataset)

Metrics return a pandas.DataFrame. You can also run the sample notebook without installing anything, in a Jupyter Lab on EDITO set up for evaluation: Open in EDITO