FAQs

Structural validation

Structural validation starts with the obvious, ensuring dimensional consistency e.g. every parameter and equation in FRIDA has an associated unit, and Stella (our modeling tool) confirms for us that we never violate that consistency, making sure we are not adding apples and oranges. 

Next, as we build the model we perform a series of extreme conditions tests, making sure that when a driver changes sign, or tends towards an infinity that the output variable responds by going in the proper direction, with the proper magnitude where proper is ideally defined by the literature, or as sometimes happens based on the understanding of a group of subject matter experts. 

Once the model developers have demonstrated dimensional consistency, extreme conditions sanity, and reasonable (ideally literature-checked) parameter estimates that allow the model as a whole to reproduce history, we then double check that the processes that give rise to that behavior match with what a subject matter expert (and the literature’s) conceptualizations for how those same real-world processes function. This is the “behavioural validation” part. There have been many occasions where there was model structure which was dimensionally consistent, passed extreme conditions tests and generated accurate projections of history, but was deemed insufficient because a certain key process was not accounted for. 

Behavioural validation

One of the biggest risks with any modeling project is that the model is not rigorously and empirically grounded in the best available data. One risks generating compelling narratives around a model that has a tenuous relationship to reality. One of the earliest System Dynamics IAMs –   Forrester’s 1971 World Dynamics – was accused (erroneously) of exactly that (see Nordhaus, 1973).  

With FRIDA we use the laws of nature and well-established empirical relationships to derive as many relationships in the model as possible (see D1.6 to learn about how that process was employed for modeling climate impacts). For relationships where we have less established knowledge we still use the best available knowledge to generate the structure, but we set wider parameter ranges. As a part of behavioural validation we calibrate the model to observations for the historical period (1980-2024), keeping track of the uncertainty that is inherent in the model’s calibration. To that end, there are, at this point in time, 158 different time series of data that we use to constrain the parameters (this number is expanding as we continue data archeology to improve our basis for model calibration).  

Finally we measure, via sensitivity analysis, what the range of plausible outcomes (future and past) are based on the inherent parametric uncertainty (which for FRIDA sometimes represents the equivalent of structural uncertainty in “standard” process based IAMs). More on this later in the question about “How do you measure uncertainty?“

Getting to the bottom of these kinds of questions is also a key part of the work of WP4, which focuses on developing tools for using and assessing FRIDA. When introducing the model to users, we plan to have discussions on what processes we have modeled, and what processes we have not, and likely should. 

We expect that the peer review process will add to the continuous improvement of the model, as more thought is put into whether or not we’ve chosen the proper set of processes to represent.

For more on the risk of overemphasis on “pet” processes, see the previous question “What is the scope of FRIDA? And what makes it different from other integrated assessment models?” where we discuss how we choose what processes to model.

Finally as a part of our professional network, we need your eyes, your colleagues’ eyes, the community’s eyes on this work to comment and critique our choices of what processes are included and how they are implemented. What we do have within the team, are well established practices for how to include new processes, remove unneeded processes, and ensure that what we do choose to represent relevantly reflects what has been measured about reality in the past. The advantage of building a modular, relatively simple model is that we can more easily update it if new knowledge is introduced. We would much appreciate it if you would send us any comments you may have to the model (use this form).

However, just as we have included different land types (forest, cropland) in the Land Use and Agriculture module, and different economic types (laborers, owners) within the Economy sector, we can introduce other, potentially rivalling, agents in FRIDA. Take the example of forestation:

FRIDA can represent the impact of that “global decision” in aggregate, by treating global forestation as the sum of all of the forestation happening at the regional level (which is the sum of all of the forestation happening at the country level, which is the sum of… ad infinitum).  Many regional differences have this attribute, and are therefore dealt with via straight addition based aggregation. One would rightfully argue that it is not possible to use this same approach for every possible regional process, and that would be correct. In some cases an averaging technique is used, or in others a median, and sometimes in other cases, a more complex statistical relationship is extracted directly from gridded data (again, see D1.6 for a description of how this was done for the case of climate feedback) but in each case, for a process that we represent which fundamentally plays out at the local, country, or regional level we aggregate the effects of that process by one of these various techniques, and keep track of a plausible range for each resultant parameter that controls the behavior of that relationship. These parameter ranges allow for the uncertainty not only in the underlying process itself, but also in its aggregation to the global level. 

One of the side benefits of our focus on aggregation is that oftentimes the process uncertainty is larger at the local level than the combination of process uncertainty and aggregation uncertainty is at the aggregated global level because at the global level there is often better data that we can use to constrain the boundaries of such processes. Though we would be remiss if we did not note, as with every decision we make about scope, there are tradeoffs. In this case each decision we make to aggregate gives us clarity of understanding, but takes us that much further away from true real-world complexity. So a tradeoff we weigh constantly is one between including more specificity, vs maintaining a simpler, more discoverable and communicable explanation.  

If FRIDA is to yield insights for the policy making process, it is not through exact values, but through the intuition, scenarios and thinking generated by using FRIDA. Such insights must be further investigated and refined in regional and sectorally specific tools to yield the “exact values”. For us the primary scientific aim of FRIDA is establishing new ways of thinking about climate feedbacks, demonstrating new insights/understanding which arise from taking the feedback perspective and focusing on scenario coherency and consistency above all else.

But to answer the question: if the process is not explicitly modelled then it is implicitly assumed to continue as it has during the calibration period. 

“ScenarioMIP requests that the IAM teams produce simulations that do not include climate change impacts on managed systems (e.g. agriculture, energy use, or economic growth). At this point in time, there are two main reasons for this. First, one of the main uses of the scenarios and their climate outcomes is to drive impacts estimation by the impact modeling community, which uses both the climate projections and the direct human drivers (such as land use and agricultural systems changes) as input to their analyses. If the IAM scenarios (and therefore the climate projections based on them) already include impacts, further impact modeling based on these scenarios would lead to double counting. Second, IAMs currently do not represent a full range of possible impacts and generally lack the required detail needed to represent many regional impacted systems and adaptation strategies. Including impacts in the IAM scenarios would therefore only provide a partial and somewhat arbitrary accounting of possible climate effects. The IAM and ESM scenarios are therefore not intended to provide complete pictures of potential future worlds. Rather, they must be augmented by impact and adaptation studies that complete that picture so that it includes climate shifts, mitigation, impacts, adaptation, and development. At the same time, demand for fully consistent scenarios is growing. It is, therefore, encouraged that IAM modelers undertake research projects to produce additional scenarios in which impacts are accounted for. This work may also lead to different scenario protocols for future ScenarioMIP exercises.”

The  “standard” process-based IAMs are what is described before the bolded sentence. FRIDA is one of the set of models to respond to the encouragement of the bolded statement: to produce fully consistent scenarios. To do so we need to include relevant feedbacks from climate back to the human side of the system. Almost all models contain numerous feedbacks, and FRIDA contains a massive amount of them. But why are they important for producing consistent scenarios?  The explanation is relatively simple: human and climate processes do not exist in a vacuum, fully divorced from each other, so we need feedbacks from one to the other. As we all well know, actions taken by humans have impacts on climate, for example the combustion of fossil fuels to produce energy. Likewise, the behavior of climate has impacts on humans, e.g. a wildfire in Los Angeles (caused either by human processes, or climate process, or some combination of the two) destroys lives, homes, and businesses. So from the position of reality it is impossible to argue that feedback from climate to humans can be ignored. Instead, all one can argue is that a certain feedback isn’t relevant to a particular scenario narrative, or a particular period of time, or a particular place. To produce realistic and therefore consistent projections of the future evolution of the combined human and climate systems we need to start from the feedbacks that tie the human and climate systems together.

For the first we add or remove feedbacks one by one in the model and investigate the resulting impact on a set of key variables in the model. This is work in progress by one of our postdocs. For the second, although we will make such comparisons in practically all our publications, they will likely not be measures of the importance of climate feedbacks, simply because there are so many other differences between the various models. 

As Jochem Marotzke pointed out, there is seldom such a fixed-frequency oscillation in nature (if one disregards astronomical phenomena). Missing from the uncertainty analysis done in January were key parameters which controlled the period and amplitude of the oscillation. Once we included these parameters in the analysis the oscillation disappeared and we thank the council for the push to include all model parameters into the uncertainty analysis.  

Here is the old (left) and corrected (right)  figure 10a (GDP). In the updated ensemble mean on the right hand side the oscillations do not have a constant period or a large amplitude. The individual runs do have strong oscillations, but these cancel out in the mean since the frequency is not constant.

To get to the bottom of the story of the oscillations we need to look at how climate change impacts the economy. 

One process which comes up time and time again in our discussions of climate impacts, and the co-evolution of the combined human and climate system, is the role that climate change plays in economics, and specifically on finance. We have developed our own representation of this feedback in a process called “failure rate of loans”. The gist of this process is that as the surface temperature anomaly rises, and extreme events become more prevalent, the environment in which businesses operate changes, causing a marginal rise in the default rate for loans. In response, banks raise their lending standards, reducing failure wrought lending to maintain profitability. This reduces the default rate, but has the side-effect of reducing investment, a critical driver of economic growth, and the cycle (an oscillation in this case, one of the primary oscillations within the model) continues.

How does this work? We represent banks as a profit making entity, generating loans (“bank investments”) in order to turn a profit for their owners. The bank’s current lending standard is a state variable in the model. Some of these loans will fail whereas the rest will succeed and will be repaid in full with interest. We assume that the banks do not know which is which; from the perspective of the banks there is one pool of “risky” loans. The rate at which loans are classified as risky is determined by three variables: 1) climate (which for the purpose of this description we assume is a null effect), 2) the banks’ lending standards (the higher the banks’ lending standards, the lower the current failure rate), and 3) the ratio of growth in investments to the ratio of growth in overall GDP (when investment growth surpasses the economy growth potential the ratio of bad investments increases, and vice versa).  

According to observations, bad loans default on average approximately 10 years after the original loan was made. The globally aggregated bank does not react immediately though, there is a reporting delay of approximately half a  year before the globally aggregated bank can compile the information about defaults across the entire world, and with that information we assume that the globally aggregated bank adjusts their lending standards. Let’s say in this hypothetical case the measured default rate is higher than expected, and therefore lending standards are raised. Those higher lending standards do two things. First they immediately reduce the failure rate, because now that banks are being more scrupulous with their investment, they’re better able to weed out the failures ahead of time. Second, they reduce lending growth. As a result of these actions, after a delay, the default rate drops, and now banks lower their now “overly high” lending standards, increasing their rate of loan creation and their ratio of bad loans, restarting the cycle all over again. In this cycle the period of the oscillation is controlled by a combination of the default time, and the reporting delay. The amplitude of this oscillation is caused by the rate of over-correction in lending standards made by the bank. The bigger the over-correction, the larger the amplitude, the longer the delay, the longer the period. These specific values for these variables were determined during calibration when fitting the model to historical data on investment, consumption, government spending, wages, employment, nominal GDP and real GDP. 

This oscillation is one of a handful of powerful, structure driven, oscillations in the economy module, and because of the feedback complexity, the two-way coupling of climate and humans, those oscillations show up across the full FRIDA model. As for the realism of these oscillations, there is plentiful evidence (see a list of references in D2.5; see also Martinez-Moyano et al., 2014).  But they are not as regular as would be implied by FRIDA over a single run, because there are many perturbations from that regular cycle that come from processes not modeled which disturb the regularity of the oscillations (e.g. shocks, and other things outside of the scope of FRIDA).

It is true that at this point we do not have a complete and polished written explanation of everything that happens in the model, but we are entering a period of model analysis this spring, now that we have a publishable version of FRIDA (v2.1) and a large number of questions that the team is in the midst of investigating. For our “tools for using FRIDA” (WP4) it is paramount that we create understandable answers not just to “what happened” (when we pulled a lever), but also “why did this happen”. We acknowledge that future trust and acceptance of FRIDA will hinge largely on our ability to quickly convey why things are happening in the model.

We also use parametric sensitivity analysis – i.e. touch individual parameters one at a time, and see how changes develop based on the magnitude and direction of the change. Other methods include carefully upgrading the model, using an almost religious devotion to the idea of small atomic changes. Introduce one single relationship at a time, study how the introduction of that relationship changes system behavior then add the next small relationship, keeping track of the changes caused by a larger change as the accumulation of impacts from smaller changes (for instance this is how we know about our finance driven oscillations, and this is validated by the economic literature on the subject, see Schumpeter & Minsky, summarized e.g. by Knell, 2015). There are also a variety of other analytical methods which could be applied to parts of FRIDA to measure the origins of behavior and on the right occasions these are employed – but generally they are hard to apply because of the relatively “large” size of FRIDA compared to other models with simpler levels of dynamic / feedback complexity.The novel method at our disposal is Loops that Matter (LTM; see Schoenberg, Davidsen and Eberlein (2020)). LTM is an automated method for doing formal loop dominance analysis. What that means in plain English is that Stella contains algorithms which are able to walk the network of equations that constitute the FRIDA model, analyzing for each equation how much a change in each independent variable creates change in each dependent variable at each and every time step. Those link gains are then multiplied together to create loop gains, and those loop gains are then normalized across all feedback loops in the model giving us percentage scores which report at each and every dt the importance of that feedback loop to the change in model behavior observed at that exact point in time. We can run this analysis with different combinations of policies, or calibration parameters to discuss how feedback importance changes under different conditions or sets of action. The trouble with this approach is that in a model that has the level of feedback complexity that FRIDA has, the results of that analysis are not so simply understood and communicated. While the distribution of feedback importance in models like FRIDA tends to be heavily skewed (i.e. out of many loops only a very small few are explanatory above a lets say 1% threshold in any one scenario across all time periods) the interpretation of the loops discovered can be difficult to communicate, as our results have shown that it is not the short, intra-module feedbacks that dominate the behavior of the system, but instead the long, inter-module feedbacks that matter. What often ultimately ends up being more communicable isn’t necessarily the full feedback loop description, but the description and identification of an important smaller piece of structure that allows one or many full feedback processes to be dominant across multiple periods of time. It is on our critical path (for October 2025) to write a report summarizing the most important feedback processes in FRIDA, and how those processes interplay to create different outcomes. Our plan for producing this report is to use a combination of traditional methods, and LTM to generate that understanding.

In the future, we will also introduce stochastic behaviour in the climate response which will also necessitate the ensemble approach we have already embarked on. This ability to include stochastic behaviour is a unique strength of FRIDA, made possible by the model’s efficiency and structure which allows us to run millions of simulations in a tractable amount of time. The process by which we have so far run uncertainty analyses is described in D3.5 section 3 (see Figure 9: the EMB ensemble will likely remain on the order of 100,000 to 500,000 runs (we have found that the confidence intervals do not change as we add more runs than that to the ensemble)).

Some among you questioned for instance why our population uncertainty bands are so small compared to the SSPs, and why our population projections are relatively high as compared to the SSPs, guessing that we did not incorporate structural uncertainties, therefore potentially mis-representing the likely ranges for global population in the future. Our first response to this argument is to somewhat agree – in our uncertainty ensemble presented in D3.5 we did not substitute different models for fertility (which is a key driver of future population change). Instead we varied the strength of all of the relationships that we have modeled which drive fertility such as female education, and economic output per capita. In addition we also varied all of the parameters which control death rates over their plausible range, while constraining parameter values according to past behavior. 

After the Portugal meeting we went back to the demographics module, and made an adjustment that varies future fertility over a wider range that is more in line with literature.  As a result our current population projections are far wider than they were in January (compare the left hand side, the results shown in Portugal, with the right hand side, the updated results, in the figure below).

But are our population projections erroneously wrong? We argue no. First of all, many of the SSP quantifications (shown in D3.5) fall outside UN data for global population, many are low compared to today’s population, as well as modern UN projections. 

Secondly, it should be noted that population (especially fertility) is closely related to economic output, so it is to be expected that FRIDA, which has much lower GDP growth than most IAMs due to the inclusion of climate impacts, will by necessity have a relatively higher population growth.

Given the knowledge of past behavior (UN population data by age group through 2023) and the structure of the demographics model (plus the structure of the rest of the model which controls the drivers of demographics) we believe our projections to be structurally valid.