IAM COMPACT results validation and vetting app released!

IAM COMPACT has developed validation and vetting checks to be applied to model results that are submitted as part of modelling studies throughout the project. Validation here refers to checking reporting formats, naming conventions, and self-consistent aggregations, while vetting refers to checking results against certain expected reference values. The project has developed Python packages that can be used to carry out the checks locally on the computers of modellers and/or modelling study leads/coordinators, and it will be integrated into the I2AM PARIS platform and applied to IAM COMPACT modelling results uploaded therein. 

Validation and vetting app

https://validation.iam-compact.eu/ 

The validation and vetting app is used for performing name-validation and region-mapping on the modelling results of Integrated Assessment Models (IAMs) in the IAM COMPACT project, as well as vetting using the vetting criteria from IPCC AR6, and comparison with harmonisation data (at the moment, only population and GDP). Harmonisation data can be found in D4.4 – Broad Scenario Logic update[1]. Specification of the vetting criteria for IPCC AR6 vetting rules can be found in the IPCC AR6 Working Group III report, Annex III, Table 11 (see here).

Validation of model outputs

The app is built in Python and Streamlit and will be eventually integrated in I2AM PARIS, an open-access, data exchange platform for modelling results (link). It is based on a fork from the I2AM Paris validation app. Installation instructions can be found here.

The ‘iamcompact-nomenclature’ package builds on the IAMC packages ‘nomenclature-iamc’[1] and ‘pyam-iamc’[2] for the code to perform the validation checks and uses the repository ‘common-definitions[3]’ of standard IAM variable names as the starting point for the list of variable names. It then supplements with additional variables from the openENTRANCE project[4], where ‘common-definitions’ lacks an appropriate variable, or in some cases from the variables used in the IPCC AR6 WGIII scenario database hosted at IIASA[5] for backwards compatibility. 

Vetting tools and criteria

While validation ensures that model results are reported in the proper file formats and using correct naming conventions, as well as guards against spelling errors and other easily identifiable mistakes, it does not catch errors or values that should be investigated further in the numerical results themselves. Examples of such errors or numbers needing attention are variable values that are not consistent with harmonised input or calibration data, energy values that do not use the correct convention for computing the primary energy of non-thermal power sources, or variable values that are outside reasonably expected ranges. 

In IAM COMPACT, the term vetting is used to refer to the process of flagging possible errors in the numerical data, or of identifying variable values that otherwise should be investigated further. It does not mean a process of filtering out and excluding model results based on deviations from some predetermined expected ranges. No such filtering or exclusion is done at the project level in IAM COMPACT. Rather, it is up to individual modelling studies whether they want to establish any criteria for inclusion, based on the research questions being addressed. 

As in the case of validation, a Python package, ‘iamcompact-vetting’, has been developed[6] to perform the vetting checks. Python functions in the package take model result files in IAMC format as inputs, and report results and the magnitude of any deviations from the reference data as output, for further follow-up by individual modelling teams and study coordinators. At the time of writing, the package was being tested on the result files from the 1st PRM cycle, and results will be reported to modelling teams when ready. The package can be used standalone locally on a PC from a Python interpreter or notebook; however, it is also planned for integration into the web-based tools on the I2AM PARIS platform.

Video demonstration

A step-by-step video demonstration of how to use IAM COMPACT’s results validation and vetting app is available here.

Important!: the tool is designed specifically for IAM COMPACT for now, and will not currently work for other projects, but feel free to check the code or the app for inspiration, or get in touch with our colleague Jan Ivar Korsbakken (jan.ivar.korsbakken AT cicero.oslo.no).

IAMC2024 Poster

The tool was also presented as a poster to the Seventeenth Annual Meeting of the Integrated Assessment Modeling Consortium (IAMC), and you can download it here.