Of Replication and Psychological Resilience: How Resilient Are Human Beings?

How resilient are people to the loss of a loved one, a natural disaster, a terrorist attack, or some other dreaded experience? The question is obviously an important one, and for a long time, it was assumed the answer was: not very. Partly because clinical populations predominated in early research, it appeared that most people took a long time to get over significant adversity. We now know that is not true. We know it on the basis of a substantial literature examining the aftermath of many aversive events (from terrorist attacks, military deployment, and mass shootings to bereavement, breast cancer surgery, traumatic injury, and heart attack).

Most people get through these experiences. They may experience some initial relatively short-lived distress, but they return to their previous functioning pretty soon after, remain able to experience positive emotions, and retain the capacity for generative experiences. More than anyone else, George Bonanno, a professor at Teachers College Columbia University, has outlined these capacities in an extensive and highly-cited body of work. He has overturned a number of wrong-headed assumptions about people’s vulnerability to potentially traumatizing experiences and the ways we cope with them.

How Common is Resilience?

Nevertheless, a recent paper takes square aim at Bonanno’s work by seeking to “replicate” two recent papers demonstrating the prevalence of resilient outcomes. (Full disclosure: I am a longtime friend and collaborator of Bonanno’s, the first author on one paper, and the third author on the other. He had no part in writing this blog post.)

Published in Perspectives in Psych Science, this recent paper—by Frank Infurna, an assistant professor at Arizona State, and Suniya Luthar, also a professor at Arizona State and a well-known scholar of childhood resilience—is entitled: “Resilience to Major Life Stressors is Not as Common as Was Thought” (Infurna & Luthar, 2016). As it suggests, they maintain that previous estimates of resilience (and the entire resilience literature) are wrong.

Are they right?

Replicating Previous Work on Resilience

First, some backstory. A firestorm has been raging over psychologists’ routine research practices, leading to the concern that many published research findings are actually false. The match that lit this firestorm was Daryl Bem’s now famous (or infamous) paper on ESP, and the gasoline that turned it into a five-alarm warehouse fire was a large-scale effort to systematically replicate 100 findings in well-known journals. The sobering results (only 38% clearly replicated) made the front page of The New York Times—no mean feat—and went on to generate controversies of its own. See here and here for debate regarding the conclusions of that study.

As these issues were thrust to the fore, replication became a cause célèbre, making research superstars out of its proponents and fueling movements toward openness and transparency in science more broadly. Few would question that these developments were overdue and salutary for psychological science. Nevertheless, in this context, Infurna and Luthar’s paper had particular pop and sizzle.

Indeed, their first sentence seizes the replication zeitgeist by the shirt collars: “Recognizing the importance of replicating major findings in psychology, we focused on recurrent claims about resilience.” They claim that previous estimates of resilience have been vastly overstated, and use a reanalysis of data we previously published to support their claim (see here and here). Their findings appear to show that less than half are resilient to spousal loss (47%); about a third to divorce (36%); and less than half to unemployment (48%).

I will argue that their conclusions are based on:

  1. questionable statistical decisions in their trajectory analyses;
  2. peculiarly strong inferences from findings that were, in fact, extraordinarily muddled;
  3. an unclear operational understanding of resilience itself.

I would also add that their study is not a replication. Instead, it is a reanalysis, as my co-authors explain here.

Methods for Determining Resilience

There are a lot of ways of determining rates of resilience. One technique that has gained a lot of currency is latent growth mixture modeling. This technique groups people who experienced adversity into different reaction patterns over time and provides a rough estimate of resilience and other reactions. These estimates, like all estimates, are subject to uncertainty. They are susceptible to model decisions and to covariates; and, like all research, to sample limitations and questions of external validity. But when done carefully, they do provide a reasonably sound index of how many people show, for example, resilience vs. PTSD reactions.

Infurna and Luthar conduct a trajectory analysis using large samples of people who experienced either the loss of a spouse, a divorce, or unemployment. They analyze an overlapping sample with the one in our original paper and use the same latent trajectory technique but with very different model specifications.

As someone who has published many trajectory studies myself, I know them from a technical and a conceptual standpoint. They require an unusual degree of judgment to carry them out, and a key issue concerns how you design your statistical model. Two concerns must be balanced. You need to provide an overall structure for your model that you think reflects the underlying data. But you also need to provide enough freedom so that the data are not arbitrarily forced into your model structure.

The balance between imposing structure and providing freedom is an inherent tension in trajectory analyses. There are no absolute rules for striking this balance. And trajectory models are inherently iterative. But there are two clear criteria that guide your decision making. One criterion is that a better model tends to be one that has a better fit to the underlying data (based on measures of relative fit). Another is that a better model is one that you can make sense of (interpret or align with existing theory).

How did Infurna and Luthar conduct their trajectory analysis? They opted, to an extraordinary degree, to increase the number of parameters of in their models and to estimate variances rather than fix them. That is, they opted against imposing structure or constraints. One potential problem with this approach is that the underlying data are poorly represented, and, as a result, the model doesn’t fit very well. In fact, they produced models that had demonstrably poor fit. For example, in their analysis of bereavement, entropy was .58, which would generally be considered poor. By contrast, in our bereavement analyses, entropy was .70.

Strangely, in the introduction, the authors advance strenuous a priori justifications for their model decisions, but at the same time, they also emphasize that latent trajectory analyses are inherently exploratory. If they are exploratory, why would you determine your model a priori? Wouldn’t you want to pay attention to the data before you settled on analytic decisions?

They also chose not to link the trajectories to relevant correlates or meaningful outcomes (as we did in our original paper). This would have helped to shed light on the meaning or validity of their trajectories. Instead, their analyses were based entirely on a single outcome measure, with no other data points brought to bear. As a result, the trajectories were, for all intents and purposes, meaningless (i.e., uninterpretable).

Don’t take my word for it. Let’s look at the preferred model they produced for bereavement. Below you’ll see two different trajectories of life satisfaction across the 11 year span of the study (the middle time point is the loss). According to the authors, one of these trajectories is resilience and one is recovery. (Data and syntax from the authors are here on Open Science Framework.)

infurna_traj

A simple question: which trajectory is resilience? If you don’t know the answer, you are not alone. I sure wouldn’t. The trajectories are identical.  Each shows a modest dip in life satisfaction around the time of the loss. Each returns to its baseline levels at about the same time. But one trajectory (the upper one) is labeled as resilience and the other (the lower one) is labelled as recovery. In fact, both trajectories could be called resilience (bouncing back to where one started), resulting in the Alice-in-Wonderland conclusion that everyone is resilient (100%).

Comparing Trajectory Results

Compare their result with the trajectory solution my colleagues and I identified in our original paper. Note that our trajectories are not only distinctive but also correspond to theoretically relevant patterns. We found a group that is substantially affected by the loss (blue trajectory; 21.3%), which represents a grief reaction. We found one that is modestly affected (black trajectory; 58.7%), which we called resilience. We also found a group that improves (green trajectory; 5.4%), a phenomenon previously documented and likely veridical. We also found a group that was low-functioning before and low-functioning after the loss, displaying little reaction (red trajectory: 14.6%). I leave it to the reader to decide which of the trajectory solutions makes more sense.

mancini_traj_2

How did they turn our original analysis into mush? By providing far too much freedom in their models. Their analysis stands apart from every other latent growth mixture model I have ever encountered in the degree to which they freed up parameters and allowed variances to be estimated rather than fixed. As mentioned, they make a strenuous argument on behalf of this decision. But ultimately, when the results make little sense, there is little support in the literature for deciding to forge ahead anyway. See Muthen in particular on this.

Expansive and Unwarranted Conclusions

Nevertheless, the authors draw expansive, unwarranted conclusions. “Considered collectively, these factors underscore the inadvisability of any declarations on rates of resilience.” But the title of the paper is “Resilience is Not as Common as Was Thought”? You can’t have it both ways.

They also argue that resilience is not distinct from recovery. “From a scientific perspective, our findings underscore the need to reexamine not only conclusions that resilience is the modal response to adversity but also that it is distinct from recovery.” The authors’ own analytic decisions played a strong hand in producing this result. However, this possibility is not acknowledged as a potential limitation of their approach.

Further confusing matters, if the latter conclusion is true—that resilience and recovery are not distinct—then the primary conclusion that resilience is not common is without foundation. 

Scientific Responsibility

In short, the authors make sweeping claims—and sweeping dismissals—on the basis of results that border on meaningless. They use single variable analyses of bereavement, divorce, and unemployment to reject conclusions supported by dozens of studies using different methods, different samples, and different traumatic events.. They use debatable procedures, and at the same time, they exhort others to employ their procedures, without considering their limitations or that they are at variance with other published work.

They even question the scientific responsibility of previous resilience researchers: “We argue, respectfully, that resilience researchers must hold themselves to the highest standards of scientific responsibility, ensuring great circumspection in conclusions drawn on the basis of particular data sets, measures, and analyses.”

Scientific knowledge is always in flux, and scientific responsibility must be borne by all. It remains possible that resilience is not common. But this paper does not demonstrate it.

References

 Bonanno, G. A. (2004). Loss, trauma, and human resilience: Have we underestimated the human capacity to thrive after extremely aversive events? American Psychologist, 59(1), 20-28. doi:10.1037/0003-066X.59.1.20

Galatzer-Levy, I. R., & Bonanno, G. A. (2016). It’s not so easy to make resilience go away: Commentary on Infurna and Luthar (2016). Perspectives on Psychological Science, 11(2), 195-198. doi:10.1177/1745691615621277

Galatzer-Levy, I. R., Bonanno, G. A., & Mancini, A. D. (2010). From Marianthal to latent growth mixture modeling: A return to the exploration of individual differences in response to unemployment. Journal of Neuroscience, Psychology, and Economics, 3(2), 116-125. doi:10.1037/a0020077

Infurna, F. J., & Luthar, S. S. (2016). Resilience to major life stressors is not as common as thought. Perspectives on Psychological Science, 11(2), 175-194. doi:10.1177/1745691615621271

Mancini, A. D., Bonanno, G. A., & Clark, A. (2011). Stepping off the hedonic treadmill: Individual differences in response to major life events Journal of Individual Differences, 32(3), 144-152. doi:10.1027/1614-0001/a00004

 

 

 

Advertisements

Author: Anthony Mancini

Pace University Psychology Professor; trauma and resilience researcher.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s