Uneven representation of scenarios in IPCC report can lead to biased recommendations

Key findings in Intergovernmental Panel on Climate Change (IPCC) reports, such as emissions reductions in 2030 or year of net-zero emissions, play authoritative roles in informing climate policy and guiding action. These findings are based on a scenarios database. The database consists of numerous emissions scenarios that have been generated by Integrated Assessment Models (IAMs), which are tools to provide policy-relevant insight across disciplines into climate change mitigation. 
But submitting scenarios to the scenarios database is voluntary and there is an uneven representation of models and studies. Two thirds of the emissions scenarios in the IPCC Sixth Assessment Report (AR6) scenarios database come from only four IAMs, despite submissions from over 50 models. About half of the scenarios come from a single study. The majority of the models come from European institutions, with the second largest category being the US. 

Fingerprints

In a new paper published in Nature Communications, Influence of individual models and studies on quantitative mitigation findings in the IPCC Sixth Assessment Report, CICERO-researchers Ida Sognnæs and Glen Peters analysed the impact of the uneven representation of models and modelling studies in AR6 on key findings presented in the Working Group III (WGIII) Summary for Policymakers (SPM). Models and studies that have a lot of scenarios in the database have a substantial influence on findings in the SPM, the study found. 
Models have distinct ‘fingerprints’, using different strategies for reaching the same climate target. And scenario outcomes can be dependent on study assumptions. Therefore, an uneven representation of models and studies in the scenarios database may shift findings towards certain model fingerprints and assumptions. The new study is the first to quantify this effect. 

Many scenario findings in the SPM change substantially when a single model or study is excluded:

• Median  greenhouse gas (GHG) reductions in 1.5°C scenarios by 2030 — a widely recognized target, used in the 2022 Sharm el-Sheikh Implementation Plan — changes from 43% to 50% (relative to 2019) when the single model with the most scenarios is taken out. 
• Median 2030 co₂ reductions shift from 48% to 56% and median coal and gas reductions in 2050 shift from 95% to 83% and 43% to 29%, respectively. 
• The median net-zero GHG year shifts from 2098 to 2086 when removing the model with the second most scenarios, from 2098 to 2084 when removing the study with the most scenarios, and to after 2100 when removing several other models and studies.

This means that, had the number of scenarios submitted to the IPCC database by different models or studies been different, headline mitigation findings could have been different.

Median greenhouse gas emissions reductions in IPCC scenarios that limit global warming to 1.5°C. The orange dashed line shows the IPCC median and the blue line shows the median when the model with the most scenarios is taken out.

– It is the model with the most scenarios that has the largest influence on 1.5°C scenario findings. Individual studies have only a small or negligible impact on most findings, said Sognnaes.

This is partly because the dominant model, REMIND, is responsible for a much larger share (42%) of the 1.5°C scenarios, than the dominant study, ENGAGE, (26%). Reported emissions reductions by 2030 are lower because REMIND reduces emissions more slowly in the near-term compared to other models. REMIND reduces coal and gas more heavily than oil, and this ‘model fingerprint’ is seen in the statistics in the SPM. The only finding that is significantly impacted by the dominant study is the net-zero GHG year. The ENGAGE study has a large influence on the reported net-zero GHG year because many ENGAGE scenarios do not reach net-zero GHG emissions before 2100 by design.
Scenario outcomes are often compared across climate categories to show the implications of different climate targets, for example 1.5°C with and without overshoot. But the study shows that different climate categories are dominated by different models. This means that differences in scenario outcomes may be more reflective of differences in model sampling than of the climate target. 

How can this problem be avoided?

- The purpose of scenario analysis is not to provide precise estimates of individual scenario outcomes, but to show the implications of choices and trade-offs. The IPCC assessments should focus more on how scenario outcomes depend on the assumptions that are captured by different models and studies. The robustness of statistical findings should also be assessed against the uneven sampling, said Peters
The current use of descriptive statistics to present key scenarios findings gives a lot of weight to the subset of scenarios that are submitted to the database, and ignores scenarios not submitted to the database. But the IPCC is meant to assess the full scenarios literature independently of how many scenarios are submitted to the database.
Even if the IPCC scenarios database contained all the scenarios from the literature, however, the use of single variable descriptive statistics may still not be the best way to reflect the insights contained within the scenarios literature. The mixture of different research questions and assumptions across different studies means that descriptive statistics from the database are simply not very meaningful. 
– Modelling is for “insights not numbers” and scenario outcomes carry little meaning when they are not interpreted with respect to the assumptions under which they were generated and the research questions they were designed to answer. To fully capture the meaning of scenario outcomes, there is no way around assessing the original research papers, said Sognnaes.

Moving forward

The study confirms that different models provide different views. The use of simple database statistics to present key findings in IPCC reports might mean that targets and decisions are biased by models that happen to have a lot of scenarios in the database.
Scenario databases have a diversity of uses, but there might not be a one-size-fits-all method for analysing databases. A transparent discussion of how to leverage databases for different purposes, contexts and outputs is important. 
Moving away from descriptive database statistics that are difficult to interpret and sensitive to sampling might be more in line with both the goal of the IPCC, to assess the full scenarios literature, and the purpose of integrated assessment modelling, to provide insights not numbers.