Measurement quality assurance prepares the raw data for further analysis. Data filtering, gap filling, and homogenisation add significant value to raw measurement data.
meteoblue provides measurement quality assurance based on the measurements available via the API. meteoblue distinguishes between three different use cases: single station, multiple stations and city network. The quality assurance is provided for the following weather variables:
- Air temperature
- Wind speed
- Wind direction
- Solar radiation
- Relative humidity
The measurement quality assurance allows for the tracking of data transmission through real-time alerts in case the stations are not sending information. Moreover, we create monthly reports and meteorological statistics for climate indices and station coverage. In addition, the system provides data normalisation in equally distributed timestamps, and offers functions such as station quality control, gap filling, and data homogenisation. The availability of these functions depends on the use case and the meteorological variable.
Equally distributed timestamp
Different station providers can record the measurements with different temporal resolution. To compare the measurements of different station providers, it is necessary to compare the same timestamp. Additionally, the stations typically do not record exactly during the full or half hour. Therefore, the timestamping tool interpolates towards the full and half hours. The user can decide on the temporal resolution of the dataset. If the selected temporal resolution is smaller than the temporal resolution of the station provider, the values in between the regular temporal resolution are filled with NaN (Not a Number).
Station quality control
Quality control is conducted as a three-step process:
Metadata of the stations are checked for correctness. If the latitude or longitude of the station is located outside the natural coordinates of the Earth, the station data is submitted to a correction process. Additionally, the data coverage is calculated, based on the ratio of the relevant values (i.e., those which are not equal to NaN).
Static quality control for each location separately
Several different filters are applied to the time series of the individual station. These filters are typically different for different variables. The individual filters and the respective thresholds of each variable are described in more detail in the technical documentation. (link when ready)
|QC1 filter for temperature||Description|
|NaN check||Detect NaNs|
|Logical check||Detect illogical values that are too high or too low|
|Static check||Detect outliers outside of thresholds|
|Flatliner check||Detect flatliners where consecutive values are the same|
|Blip check||Detect sudden temperature changes above threshold|
|Spatial check||Detect spacially illogical values|
Dynamic quality control
Dynamic quality control is used for complex comparisons between different stations (only applies for the use case of city network, see above). The spatial consistency of the stations is verified by applying neighbouring checks, and comparing stations within a specified radius around the station.
Three different approaches are utilised for the gap filling.
- Linear interpolation
- If the gap is only one timestamp
- If the gap is larger than one timestep
- Find the station with the best correlation based on historical measurements
- Use the best correlated station and apply bias correction (to account for a potential bias between the two stations)
- If the gap is larger than one timestep
- Use model data from the meteoblue model chain which best fits the station (selection of up to 20 different numerical weather prediction (NWP) models depending on the location)
The meteoblue data homogenisation tool contains several functionalities for a variety of use cases.
For temperature measurements, two different approaches (bias correction and radiation correction) complement each other and are available based on different assumptions. In case a low-budget city sensor network is used, a bias correction can be applied in the operational and historical use case if at least one low-budget sensor is installed close to a standard WMO station (e.g., METAR, MeteoSwiss, etc.). The bias correction calculates in real-time the difference between the low-budget sensor and the WMO station, and corrects this difference for all the sensors in the city network with an update frequency of the WMO measurement. This bias correction is applied to all variables available in the city network. This method accounts for systemic discrepancies between the WMO station and the city network, e.g., errors due to heat accumulation or systematic errors occurring during nights.
For the use case “historical city network” a scientific radiation correction is applied to ensure the best possible quality of historical weather data. Note that due to the complexity of this radiation correction, it is not possible to provide this method in operational usage.
Therefore, results in operation differ from results in the historical data package, where typically the highest quality is reached.
The scientific radiation correction is based on our special routines published in a scientific research paper, and includes separate corrections for shaded and sunny conditions. Direct shortwave radiation, diffuse shortwave radiation, air temperature, wind speed, and the solar elevation angle are used as input variables for the radiation correction.