The following is a summary of various studies based on precipitation sums. These include the verification studies from 2017 and 2019, the provider comparison analyses and additionally, the comparison of various models of the NEMS model family.
Global verification study 2017
Throughout this study, the meteoblue MultiModel was compared with other forecast models like ERA5, GFS, CHIRPS2 and NEMS. 6505 METAR precipitation measurements of the year 2017 (world widely distributed) were used to verify and compare the forecast of these models.
The model performance of the total annual precipitation in 2017 can be significantly increased by combining two (or more) models of the forecast data set (see figures below). MFGLOBAL, GEM and GFS overestimate the total annual precipitation, whereas NEMSAUTO and ICON underestimate the total annual precipitation. A positive and negative bias of raw models could lead to a significant increase in the model performance by combining different models. More than 200 different model combinations represent a better global MBE than the best raw model (GFS). Also, they represent a better MAE than the best raw model (ICON). Hence, the systematic bias could be reduced to 0 and the MAE reduced to values below 170 mm. The correlation coefficient for the best model combinations is above 0.71 for the total annual precipitation, whereas the correlation coefficient of single raw models largely varies from 0.53 (GFS) to 0.67 (ICON). The best model performance is typically reached by combining more than two models. However, when combining more than two models, the performance of single events (e.g., daily precipitation > 1 mm) is typically decreasing. Therefore, we calculated the best model combinations with only two different models separately.
Global verification study 2019
Precipitation measurement data of 8112 WMO stations were compared to the 24h precipitation forecast of different models (ERA5, NEMSGLOBAL, GFS05, MFGLOBAL) for the year 2019 (Bachelor thesis Huonder, 2020). The following results refer to precipitation sums.
The results from 2019 match very well with results from 2017, indicating that GFS and MFGLOBAL tend to overestimate yearly precipitation sums, whereas NEMSGLOBAL tend to underestimate yearly precipitation sums (Table below). The lowest MAE (hence the most accurate model) is the reanalysis model ERA5.
|MAE [mm]||MBE [mm]||MPE [%]||RMSE [mm]||SD [mm]|
For the year 2019, a detailed precipitation verification for different months was conducted, indicating seasonal dependencies of the model accuracy of different models (Table below). For all months, ERA5 performs very well, with MPE constantly below 10%. The largest errors throughout the year were found for GFS and MFGLOBAL in the late northern hemispheric spring. The highest errors in NEMSGLOBAL were found in July and August, whereas the errors tend to be smaller in the northern hemispheric winter.
Finally, for each country, the best model performance for yearly precipitation sums was analyzed. The analysis shows that no clear patterns can be observed (Map below).
The competition analysis between meteoblue forecast and different competitors allows us to evaluate the performance of our forecast in the market. 500 world wide distributed precipitation measurements from GSOD allowed a comparison of the forecast of precipitation sums for the whole year of 2021. Furthermore, the precipitation forecast error for 100 locations in North America was calculated through METAR measurements from January 2021 to July 2021.
On average 362 mm over the 500 locations worldwide were recorded from GSOD (global summary of the day) for the first half of year 2021. The meteoblue forecast estimated 369mm for the 1 day forecast, which is equal to a mean bias error (MBE) = + 7 mm. The MAE for the same time period and locations is 135 mm.
In addition to that, the table below shows the errors in the precipitation sums for North America in the first half of 2021 for the meteoblue 24h precipitation forecast in comparison with 7 competitors.
Based on the updates on this study, on average 712.04 mm over the 500 locations worldwide were recorded from GSOD (global summary of the day) for the whole year 2021 and the meteoblue forecast estimated 720.46 mm for the 1 day forecast, which is equal to a mean bias error (MBE) = + 8.42 mm.
|Provider||MAE [mm]||MBE [mm]|
In addition to global weather forecast models, regional weather forecast models were additionally tested, to see potential improvements in the model performance by lowering the horizontal resolution in the model. Models of the NEMS family (NEMS12, NEMS4 and NEMSGLOBAL) were verified with data from 1605 measuring stations provided by GSOD and located in Europe for the first half of the year 2021.
The model performance slightly increases for daily precipitation sums, when using a regional model (NEMS12, NEMS4) instead of a global model (NEMSGLOBAL). However, the difference between NEMS4 and NEMS12 is not significant, rather NEMS12 performs better than NEMS4 (Figure below).
For 1644 stations in Europe, on average 314 mm precipitation were recorded by the weather stations for the first half of 2021. NEMS4 underestimated the precipitation amount by around 20%. (Average NEMS4 precipitation amount is 249 mm). The underestimation for the NEMS12 model is 14%, for the NEMS30 global model 25%. This implies that with decreasing horizontal resolution the model accuracy does not necessarily increase.