Black boxes, green swans

Researchers at ETH Zurich assessed 16 major climate risk tools. Few came out looking good.

Want Climate Risk Review in your inbox every Monday and Friday? Then please consider subscribing:

Hunting down “green swans” — the climate risks that could upend the global financial system — should be the overriding mission of firms and supervisors alike. But do they have the right tools for the job?

That’s the question posed in a new study by researchers at ETH Zurich’s Center of Economic Research. The paper represents the first concerted effort to analyse the baffling array of climate risk gizmos produced by think-tanks, universities and private companies, and to try to determine what tool features lead to top-quality and “decision-relevant” results.

It’s conclusion? Too many climate risk tools are black boxes, the outputs of which users are often unable to explain. In addition, few effectively cover all risk sources, meaning some “green swans” may evade their scrutiny.

It’s not all bad news, though. The study also identified ways in which climate tech vendors can improve their tools to produce more meaningful results.

A rummage inside the toolkits

First, though, a brief rundown of how the study was conducted. ETH Zurich’s researchers analysed 16 tools focused on measuring climate transition risks: those related to efforts to reorient the world economy to a zero carbon future. 

Source: ETH Zurich

Each was subjected to a two-tier assessment: first, a “descriptive analysis” to gauge the extent of their coverage of climate risks and the financial exposures that could be affected by then. This assessment also scrutinised tool outputs: the projected impacts of the climate risks on assets.

Second, a “criteria analysis” to determine the “quality, comparability and decision-relevance” of each tool, measured across three dimensions: accountability, depth of risk analysis, and usability. 

To best judge the 16 tools, the researchers grouped them into three categories depending on their purpose. “Climate alignment tools” are those which measure how well a certain business activity accords with a specific transition scenario, with the “gap” between the transition pathway and activity used as a proxy for transition risk exposure. “Climate impact tools” are those that assess the greenhouse gas (GHG) emissions of specific activities, providing firms with evidence of which assets and investments are contributing most to global heating. Finally, “climate target-setting tools” are intended to help institutions define transition strategies of their own and monitor their efforts to line up with them.

Each tool type has its own particular uses, strengths and weaknesses. But the study is less focused on these than each tool’s capacity to produce useful climate-related risk information.

The descriptive analysis

So, how did they do under the “descriptive analysis”? The short answer: not great. Just five of the 16 tools covered all analysed climate risk sources and risk realisations as part of their standard settings.

In addition, just five proved capable of capturing multiple sources of transition risks, and most tools fell short in recognising how transition risk impacts are “mutually reinforcing” — or in other words, how they interrelate. By failing to capture these feedback loops, the researchers said climate risk tools could understate the intensity of transition shocks.

In terms of financial asset coverage, the results were a little better. Nine tools covered more than five asset classes, and most of them covered stocks and bonds, two of the most well-traded types of security. Still, few could run climate impact analyses on mortgages or real assets, which make up a large chunk of institutional portfolios. In addition, just four tools covered options and derivatives.

Carbon4 Finance’s Carbon Impact Analytics tool had the broadest asset coverage, able to offer climate-related impact analysis for all nine individual asset classes as well as for portfolios and funds. 

Institutional Shareholder Services’ Portfolio Climate Impact Report, Ortec Finance’s ClimateMAPS and Vivid Economics’ Climate Risk Tool also covered most financial assets.

In addition, the ClimateMAPS tool offered the fullest complement of climate-adjusted financial metrics as outputs, along with the Carbon Risk Management tool developed by the University of Augsburg. PWC’s Climate Excellence risk tool also provided users with the capabilities to produce a wide range of metrics, including climate-adjusted Sharpe ratios, alpha, beta, and value-at-risk.

However, just six tools provided a “climate-adjusted firm value” — in other words a post-stress market capitalisation or enterprise value — for covered exposures. The researchers marked this as a significant shortcoming, since firm-level metrics typically underpin fundamentals-driven financial asset valuation. Though most tools offered markets-based valuation metrics, these could prove misleading so long as markets lowball their transition risks.

As relates to the climate transition scenarios employed by the tools, the descriptive analysis found that the International Energy Agency’s (IEA) scenarios were favoured by six of the 26 tools. This figures, considering the IEA models “currently provide the most granular global sector coverage”, according to the researchers. However, they added that IEA-based tools may produce skewed results since these scenarios “have been found to systematically underestimate the growth of renewable energy technologies.” Furthermore, the researchers claimed that IEA models are relatively opaque, meaning it’s difficult to critically evaluate their underlying assumptions and decision pathways. 

What should set alarm bells ringing among the users of these climate tools, though, is that some vendors appear not have a firm grasp on the key scenario assumptions used by their tools: such as the temperature peak year and the year of net-zero emissions at the country, global, sector and firm levels. This is a major shortcoming, since understanding these variables is essential to properly interpreting each tool’s output correctly.

The criteria analysis

The “criteria analysis” found that few climate risk tools satisfied the broad range of indicators for accountability, depth of risk analysis, and usability. The three tools that fulfilled, or partly fulfilled, the most were Vivid Economics’ Climate Risk Tool, the X-Degree Compatibility Model by right. (a start-up based in Frankfurt), and the E3ME-FTT-GENIE developed by Cambridge Econometrics

On accountability, just four tools performed highly, with the 2 Degrees Investing Initiative’s Paris Agreement Capital Transition Assessment (PACTA) model and Carbon Risk Management model developed by the University of Augsburg fulfilling or partly fulfilling the most criteria. 

Source: ETH Zurich

In contrast, most tools did well on depth of risk analysis, with seven identified as top performers. Oliver Wyman’s Climate Transition Risk Methodology performed best in this dimension, completely fulfilling all 14 criteria.

Source: ETH Zurich

However on usability — perhaps the most important criterion — just two tools excelled. Two did particularly poorly — the University of Augsburg’s Carbon Risk Management model and the Carbon Impact analytics suite by Carbon4 .

Source: ETH Zurich

From black boxes to glass boxes

No single climate risk tool came out the other side of ETH’s assessment covered with glory. Some excelled in certain areas but fell short in others, while most only partly fulfilled the majority of criteria indicators.

However, the researchers said the analysis should be used by each and every climate risk tool to help make them more effective. Specifically, they identified four broad areas in which they could and should improve.

First, data. One reason many tools fell short on the accountability criteria was that few made use of third-party verification for the emissions data used, or properly explained in their model documentation how they go about filling data gaps. In addition, in order to get a well-rounded view of climate risks, more data — beyond CO2 emissions data — need to be incorporated in these tools, the researchers said, including information on methane emissions and land use.

Second, scenarios. Those tools tied to specific scenarios are inherently incapable of producing an array of different transition risk impacts. These would be useful, as institutions could use them to generate probability distribution-based outputs to help them get a full appreciation of their exposures.

The best way to produce such outputs would be if climate tool vendors embraced “scenario-neutral tools” — those that can accommodate multiple models and scenario parameters. These would allow users to compare and contrast outputs and thereby “better capture the deep uncertainty surrounding future transition developments”. They’d also be better able to deal with changes to scenario modelling practices and incorporate more up-to-date scenarios as they became available. 

Third, tool approaches. Specifically, the researchers said providers should bulk out their reporting of input data and scenarios, so that institutions can trace the projected climate risk impacts to their investments back to their proximate assumptions. They also recommended providers look to expand the stable of assets they offer analysis on, and to especially focus on mortgages and real assets.

Fourth and finally, tool use principles and reporting. The researchers explained that one way in which climate risk tools could better themselves is if they “constantly update their approaches and analysis” to keep up with the rapidly evolving field of climate-related financial risk modelling. Clearly, a “one-shot analysis” of climate risks, taken at a certain point of time, using a certain set of scenarios and assumptions, would stale quickly. Providers therefore need to be always in development mode, adding in new data, scenarios, and capabilities as they become available.

This is a lot of advice to take onboard, and for some climate risk tool providers, it may mean going back to the drawing board and overhauling some of the key features of their models. Still, if the end result is a set of tools that institutions and their stakeholders properly understand, and can be confident in their use, then the whole field of climate risk management will benefit.

Thanks for reading! Why not share this post with your colleagues?


Please send questions, feedback and more to

You can catch climate risk management updates daily on LinkedIn

The views and opinions expressed in this article are those of the author alone

All images under free media license through Canva