Get started evaluating your system

Five steps to compute the OceanBench scores of a system, see them next to the published ones and submit it. The tutorial uses a sample of the GLONET system, interpolated to 1 degree, for two forecasts. It needs no credentials.

Open in EDITO Download the notebook

  1. Try the tutorial.
  2. Install OceanBench.
  3. Evaluate your system.
  4. See your scores on the website.
  5. Submit your system.

1. Try the tutorial

The next cells compute the scores of the sample. To run them, open the notebook on EDITO, where nothing needs to be installed, or install OceanBench first, see step 2.

To compute the scores of your own system, replace the content of the next cell with the code that opens your forecasts as challenger_dataset: xarray.Dataset.

# Open GLONET interpolated to 1 degree forecasts with oceanbench, for two first days
from datetime import datetime
import xarray
import oceanbench

challenger_dataset: xarray.Dataset = oceanbench.datasets.challenger.glonet_1_degree(
    [datetime.fromisoformat("2024-01-03"), datetime.fromisoformat("2024-01-10")]
)

The challenger dataset holds one forecast per first day, with its lead days and depths:

dict(challenger_dataset.sizes)
{'first_day_datetime': 2,
 'lead_day_index': 10,
 'depth': 21,
 'latitude': 168,
 'longitude': 360}

Each function of oceanbench.metrics computes one score table from the challenger dataset. The outputs below were computed with the latest version of the library, and are checked on every commit.

Comparison to GLORYS reanalysis

oceanbench.metrics.rmsd_of_variables_compared_to_glorys_reanalysis(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Missing fraction
Sea surface height (m) [sea_surface_height_above_geoid]{surface} 0.074619 0.076040 0.076083 0.077714 0.080757 0.083328 0.085179 0.088381 0.090385 0.092795 0.021021
Temperature (°C) [sea_water_potential_temperature]{surface} 0.652528 0.653926 0.698583 0.700048 0.755007 0.769546 0.826720 0.842075 0.882379 0.893457 0.021021
Salinity (PSU) [sea_water_salinity]{surface} 0.499407 0.500453 0.498298 0.496545 0.492931 0.491488 0.490539 0.486594 0.488562 0.486766 0.021021
Meridional current (m/s) [northward_sea_water_velocity]{surface} 0.126199 0.128283 0.132071 0.132466 0.136287 0.137948 0.142672 0.144310 0.148866 0.148727 0.021021
Zonal current (m/s) [eastward_sea_water_velocity]{surface} 0.129152 0.131009 0.135246 0.136868 0.140484 0.141797 0.145982 0.147507 0.150640 0.152133 0.021021
Temperature (°C) [sea_water_potential_temperature]{50m} 1.015333 1.026076 1.050361 1.061644 1.093583 1.109044 1.151639 1.171199 1.211692 1.231885 0.024334
Salinity (PSU) [sea_water_salinity]{50m} 0.239840 0.240978 0.243237 0.244817 0.247540 0.249796 0.254122 0.256705 0.261012 0.262946 0.024334
Meridional current (m/s) [northward_sea_water_velocity]{50m} 0.116580 0.117145 0.117022 0.117885 0.119597 0.121646 0.124900 0.127665 0.130423 0.131235 0.024334
Zonal current (m/s) [eastward_sea_water_velocity]{50m} 0.119282 0.119655 0.119588 0.120952 0.122718 0.124807 0.127573 0.130039 0.132855 0.134830 0.024334
Temperature (°C) [sea_water_potential_temperature]{100m} 1.144652 1.155714 1.162067 1.170667 1.188752 1.197131 1.222995 1.232723 1.261100 1.272121 0.022565
Salinity (PSU) [sea_water_salinity]{100m} 0.191754 0.192528 0.191360 0.192643 0.193588 0.194946 0.197664 0.199727 0.202572 0.204019 0.022565
Meridional current (m/s) [northward_sea_water_velocity]{100m} 0.115183 0.115403 0.114703 0.115372 0.116036 0.117877 0.120307 0.122672 0.124196 0.125504 0.022565
Zonal current (m/s) [eastward_sea_water_velocity]{100m} 0.123082 0.123225 0.122487 0.122694 0.124027 0.125522 0.126692 0.128365 0.130287 0.131541 0.022565
Temperature (°C) [sea_water_potential_temperature]{200m} 0.890901 0.895532 0.908902 0.913137 0.919519 0.928257 0.935038 0.953204 0.961626 0.977164 0.023049
Salinity (PSU) [sea_water_salinity]{200m} 0.154964 0.155454 0.155561 0.155747 0.156421 0.156945 0.158292 0.159329 0.160705 0.161252 0.023049
Meridional current (m/s) [northward_sea_water_velocity]{200m} 0.115234 0.115165 0.113671 0.113856 0.113806 0.114501 0.115447 0.116437 0.117370 0.117927 0.023049
Zonal current (m/s) [eastward_sea_water_velocity]{200m} 0.121309 0.120924 0.119132 0.119050 0.119229 0.119544 0.119767 0.120301 0.120815 0.120689 0.023049
Temperature (°C) [sea_water_potential_temperature]{300m} 0.750025 0.748564 0.755133 0.755957 0.754630 0.757930 0.759169 0.769089 0.772835 0.781045 0.022326
Salinity (PSU) [sea_water_salinity]{300m} 0.126805 0.126900 0.126468 0.126565 0.126456 0.126827 0.127144 0.128174 0.128877 0.129814 0.022326
Meridional current (m/s) [northward_sea_water_velocity]{300m} 0.111826 0.111717 0.109974 0.110087 0.110053 0.110535 0.111326 0.111929 0.112570 0.112933 0.022326
Zonal current (m/s) [eastward_sea_water_velocity]{300m} 0.115481 0.115177 0.113337 0.113242 0.113269 0.113787 0.113977 0.114227 0.114461 0.114364 0.022326
Temperature (°C) [sea_water_potential_temperature]{500m} 0.562832 0.565969 0.553682 0.558912 0.555220 0.559865 0.564283 0.572129 0.577335 0.583570 0.024125
Salinity (PSU) [sea_water_salinity]{500m} 0.095556 0.095771 0.094893 0.095325 0.094812 0.095288 0.095620 0.096367 0.096781 0.097656 0.024125
Meridional current (m/s) [northward_sea_water_velocity]{500m} 0.101695 0.101865 0.100302 0.100133 0.099879 0.100069 0.100450 0.100378 0.100718 0.100684 0.024125
Zonal current (m/s) [eastward_sea_water_velocity]{500m} 0.105540 0.105439 0.103481 0.103418 0.103196 0.103322 0.103504 0.103734 0.103811 0.103919 0.024125
oceanbench.metrics.rmsd_of_mixed_layer_depth_compared_to_glorys_reanalysis(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Missing fraction
Mixed layer depth (m) [ocean_mixed_layer_thickness]{surface} 44.358207 44.304271 45.045022 45.222902 46.781342 47.391172 48.558569 49.437008 50.383396 50.768032 0.021021
oceanbench.metrics.rmsd_of_geostrophic_currents_compared_to_glorys_reanalysis(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Missing fraction
Meridional geostrophic current (m/s) [geostrophic_northward_sea_water_velocity]{surface} 0.050886 0.051713 0.053579 0.055095 0.057013 0.059446 0.061615 0.065130 0.066309 0.069705 0.022557
Zonal geostrophic current (m/s) [geostrophic_eastward_sea_water_velocity]{surface} 0.043220 0.043578 0.045590 0.047071 0.049088 0.051034 0.053581 0.056412 0.057950 0.060844 0.022795
oceanbench.metrics.deviation_of_lagrangian_trajectories_compared_to_glorys_reanalysis(challenger_dataset)
Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10
Lagrangian trajectory deviation (km) []{surface} 10.656894 19.698372 27.875935 35.456814 42.762672 50.04023 57.209526 64.236107 71.169189

Comparison to observations

oceanbench.metrics.rmsd_of_variables_compared_to_observations(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Observations Missing
Temperature (°C) [sea_water_potential_temperature]{surface} 0.729519 0.823223 0.817659 0.793187 0.792927 0.819722 0.884886 0.910203 0.962995 0.887030 5979 119
Temperature (°C) [sea_water_potential_temperature]{0-5m} 0.624346 0.619681 0.682740 0.630820 0.637857 0.752735 0.821872 0.825867 0.813382 0.888330 7664 359
Temperature (°C) [sea_water_potential_temperature]{5-100m} 0.912026 1.084475 1.009943 1.008887 1.114126 1.006320 1.040849 1.244951 1.088723 1.266995 112836 5648
Temperature (°C) [sea_water_potential_temperature]{100-300m} 0.931663 0.952578 0.863884 0.893616 0.846869 0.944844 0.993901 0.964605 0.922218 0.974690 101095 6097
Temperature (°C) [sea_water_potential_temperature]{300-600m} 0.533929 0.552416 0.521064 0.508863 0.501300 0.579490 0.604252 0.559651 0.511964 0.591246 102359 8300
Salinity (PSU) [sea_water_salinity]{0-5m} 0.220729 0.196434 0.218186 0.278172 0.227391 0.224342 0.216814 0.238123 0.234884 0.258069 6693 180
Salinity (PSU) [sea_water_salinity]{5-100m} 0.190704 0.177970 0.195243 0.175247 0.201145 0.212075 0.219622 0.201086 0.196724 0.212497 91317 4105
Salinity (PSU) [sea_water_salinity]{100-300m} 0.125839 0.132683 0.118843 0.123539 0.116490 0.148653 0.143685 0.120711 0.126320 0.142505 80976 5597
Salinity (PSU) [sea_water_salinity]{300-600m} 0.073264 0.081169 0.074810 0.071818 0.080771 0.082681 0.087365 0.087851 0.076447 0.083926 86809 7217
Sea level anomaly (m) [sea_surface_height_above_geoid]{surface} 0.053324 0.053783 0.053746 0.056061 0.057339 0.060209 0.062376 0.063231 0.065261 0.068485 600261 26952
Zonal current (m/s) [eastward_sea_water_velocity]{15m} 0.117897 0.118349 0.124444 0.127785 0.126481 0.125074 0.127484 0.132247 0.134528 0.133564 45601 787
Meridional current (m/s) [northward_sea_water_velocity]{15m} 0.111061 0.114060 0.113605 0.115842 0.119971 0.120249 0.125306 0.122890 0.123585 0.119400 45601 787

Comparison to GLO12 analysis

oceanbench.metrics.rmsd_of_variables_compared_to_glo12_analysis(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Missing fraction
Sea surface height (m) [sea_surface_height_above_geoid]{surface} 0.026430 0.030974 0.033858 0.038486 0.045447 0.050756 0.055267 0.061102 0.063445 0.066125 0.020289
Temperature (°C) [sea_water_potential_temperature]{surface} 0.421044 0.431532 0.531871 0.543678 0.630812 0.651927 0.728916 0.756919 0.807730 0.822519 0.020289
Salinity (PSU) [sea_water_salinity]{surface} 0.149047 0.154030 0.199788 0.205910 0.243846 0.251206 0.281096 0.287221 0.313373 0.314552 0.020289
Meridional current (m/s) [northward_sea_water_velocity]{surface} 0.052394 0.071750 0.087206 0.097021 0.107578 0.116690 0.128887 0.135761 0.144139 0.147647 0.020289
Zonal current (m/s) [eastward_sea_water_velocity]{surface} 0.052766 0.072812 0.087617 0.099584 0.112085 0.121383 0.131710 0.139828 0.146375 0.150155 0.020289
Temperature (°C) [sea_water_potential_temperature]{50m} 0.720475 0.729012 0.790809 0.812893 0.887409 0.915507 0.992013 1.022422 1.088078 1.107961 0.023667
Salinity (PSU) [sea_water_salinity]{50m} 0.111815 0.112424 0.144095 0.146534 0.168451 0.171138 0.188092 0.192149 0.205398 0.207074 0.023667
Meridional current (m/s) [northward_sea_water_velocity]{50m} 0.045584 0.056352 0.066237 0.073935 0.082868 0.090591 0.100209 0.108323 0.115206 0.119772 0.023667
Zonal current (m/s) [eastward_sea_water_velocity]{50m} 0.046822 0.057693 0.067950 0.076675 0.085579 0.094024 0.103028 0.110885 0.117134 0.122324 0.023667
Temperature (°C) [sea_water_potential_temperature]{100m} 0.560666 0.578323 0.659257 0.696238 0.788971 0.832615 0.917004 0.960005 1.016460 1.048971 0.022064
Salinity (PSU) [sea_water_salinity]{100m} 0.088303 0.089195 0.112534 0.115216 0.131745 0.134669 0.148227 0.151589 0.161094 0.162544 0.022064
Meridional current (m/s) [northward_sea_water_velocity]{100m} 0.041391 0.049357 0.058644 0.065973 0.074597 0.083213 0.091624 0.099216 0.105304 0.109346 0.022064
Zonal current (m/s) [eastward_sea_water_velocity]{100m} 0.042314 0.050674 0.059832 0.067292 0.076375 0.084957 0.093215 0.100710 0.106696 0.111993 0.022064
Temperature (°C) [sea_water_potential_temperature]{200m} 0.420232 0.427802 0.499109 0.513991 0.564385 0.594452 0.642477 0.674840 0.706572 0.734199 0.022588
Salinity (PSU) [sea_water_salinity]{200m} 0.063574 0.063662 0.081797 0.082760 0.095797 0.098424 0.109242 0.111955 0.119128 0.121012 0.022588
Meridional current (m/s) [northward_sea_water_velocity]{200m} 0.032986 0.039142 0.046427 0.053126 0.060577 0.067884 0.075287 0.082136 0.087406 0.091384 0.022588
Zonal current (m/s) [eastward_sea_water_velocity]{200m} 0.032665 0.038087 0.045892 0.051891 0.059775 0.067405 0.075058 0.081022 0.085840 0.090661 0.022588
Temperature (°C) [sea_water_potential_temperature]{300m} 0.360596 0.364379 0.425061 0.436001 0.475442 0.499343 0.540027 0.567814 0.593836 0.611261 0.021799
Salinity (PSU) [sea_water_salinity]{300m} 0.056228 0.056941 0.070000 0.071388 0.080352 0.083416 0.091130 0.094394 0.099159 0.101022 0.021799
Meridional current (m/s) [northward_sea_water_velocity]{300m} 0.030394 0.035772 0.042473 0.049081 0.056181 0.063298 0.069956 0.076664 0.081354 0.085046 0.021799
Zonal current (m/s) [eastward_sea_water_velocity]{300m} 0.030076 0.034848 0.042032 0.048000 0.055183 0.062034 0.069175 0.074922 0.079310 0.083850 0.021799
Temperature (°C) [sea_water_potential_temperature]{500m} 0.233525 0.239531 0.289956 0.302122 0.346750 0.364389 0.404086 0.423508 0.451614 0.464799 0.023854
Salinity (PSU) [sea_water_salinity]{500m} 0.040848 0.041236 0.052282 0.053463 0.061255 0.062912 0.068970 0.070544 0.074994 0.076421 0.023854
Meridional current (m/s) [northward_sea_water_velocity]{500m} 0.026971 0.031315 0.037525 0.042914 0.049317 0.055287 0.061266 0.067079 0.071688 0.075160 0.023854
Zonal current (m/s) [eastward_sea_water_velocity]{500m} 0.026028 0.030597 0.037062 0.042322 0.048420 0.054218 0.060134 0.065352 0.069424 0.073173 0.023854
oceanbench.metrics.rmsd_of_mixed_layer_depth_compared_to_glo12_analysis(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Missing fraction
Mixed layer depth (m) [ocean_mixed_layer_thickness]{surface} 40.866022 40.842193 42.413561 42.315755 44.244415 44.906734 46.488266 47.098008 48.293673 48.522068 0.020289
oceanbench.metrics.rmsd_of_geostrophic_currents_compared_to_glo12_analysis(challenger_dataset)
Lead day 1 Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10 Missing fraction
Meridional geostrophic current (m/s) [geostrophic_northward_sea_water_velocity]{surface} 0.020013 0.022467 0.028176 0.032801 0.038536 0.044444 0.049828 0.056172 0.058662 0.06322 0.021344
Zonal geostrophic current (m/s) [geostrophic_eastward_sea_water_velocity]{surface} 0.018436 0.020719 0.025438 0.029535 0.034625 0.039275 0.044081 0.049488 0.051866 0.05554 0.021698
oceanbench.metrics.deviation_of_lagrangian_trajectories_compared_to_glo12_analysis(challenger_dataset)
Lead day 2 Lead day 3 Lead day 4 Lead day 5 Lead day 6 Lead day 7 Lead day 8 Lead day 9 Lead day 10
Lagrangian trajectory deviation (km) []{surface} 4.760679 9.892733 15.173815 20.528521 26.131008 32.208874 38.623199 45.215576 51.948139

2. Install OceanBench

pip install oceanbench

Evaluating 1/12° systems requires a Copernicus Marine Service account, see how to install OceanBench and what you need.

3. Evaluate your system

Write a Python file that opens challenger_dataset, like the first cell of the tutorial, and evaluate it from the command line:

oceanbench evaluate challenger.py

or from Python:

import oceanbench

oceanbench.evaluate_challenger("challenger.py")

This is the official evaluation: it computes the scores with the functions above, so the tables are the same as in this tutorial, and prints them. It also writes all the scores, with every digit, to a JSON scores file named <challenger>.<region>.scores.json. See how to evaluate a model from the command line and from Python.

4. See your scores on the website

Add the scores file to the Scores page with Add a scores file, or give its public URL, to see up to 5 systems next to the published scores. The files are read in your browser and are never uploaded. See Your scores on the website.

5. Submit your system

To have your system among the official challengers, see how to submit your system.