Here we take a look at several different perspectives on NASA’s Space Shuttle Challenger disaster. 7 astronauts were lost due to failure of a seal on the solid rocket booster 73 seconds after liftoff, which led to vehicle breakup. Two different Keepers are involved: Edward Tufte’s Visual Explanations and Diane Vaughan’s The Challenger Launch Decision. Additional online sources are referenced along the way, and Tufte’s broader contributions on communication of information are discussed.
Decades ago I took a one-day class on Information Presentation given by Edward Tufte, a professor of Statistics and Political Science at Yale. The current version is now available as online video, but I haven’t viewed it myself. Back then Tufte was a great showman, emphatic and opinionated and entertaining, with a performance built on compelling case studies of both wonderful and dreadful chartsmanship, and their impact on effective communication. Among these (and written up in the book Visual Explanations) is an assessment of information related to the launch decision for the Challenger space shuttle, which ended in tragedy after urgent warnings of dire risk. He showed that re-casting the display of history of damage, from chronology to a function of temperature at launch, completely transforms the interpretive burden for the consumer of the presentation. This case study resonated particularly strongly with someone involved in NASA mission design. I still remind (and am reminded by) colleagues that we need to “Tufte-ize” our presentations. I have used the “before and after” displays about the temperature effect on solid rocket seals in presentations on Risk Perception.
Recently I was told that Tufte misrepresented the state of engineering knowledge at the time of Challenger’s launch. I was sufficiently troubled by this claim to google “Tufte Challenger misrepresentation”, and was surprised that the top hit was a site devoted to ethics: Representation and Misrepresentation: Tufte and the Morton Thiokol Engineers on the Challenger | Online Ethics This paper, which was co-authored by Roger Boisjoly, who presented some of the charts that advocated against launching Challenger at temperatures below 53 deg F, provides valuable context regarding the uncertainty about the temperature effect on seal performance. It also indicates that Tufte did not appreciate an important distinction between erosion and blowby as two distinct physical mechanisms involved in joint degradation. The paper emphasizes that the joints had been problematic throughout the lifetime of the Shuttle program, and temperature was identified as a potential contributor relatively recently. The authors state “Those few data points were all the engineers had–despite their best efforts to get more. And they did not connect up those data points with temperature because they only suspected but did not know that cold and O-ring compromise were causally related and, as we have said, were not arguing that they were.” But even if we accept that the information state was less than what Tufte believed, there was at least a suspicion that damage could be connected with temperature. They bemoan that decisionmakers did not accept their recommendation, without re-examining the quality of evidence available to decisionmakers.
In Visual Explanations, the discussion of Challenger indulges in the use of inflammatory phrasing: “scandalous discrepancy between the intellectual tasks at hand and the images created”; “in designing those displays, the chartmakers didn’t quite know what they were doing”. But I don’t see any charges of ethical or moral failure from Tufte himself. He notes, almost in passing “we encounter diverse and divergent interpretations, as the facts of the accident are re-worked into moral narratives” before getting to his primary focus: “regardless of the indirect cultural causes of the accident, there was a clear proximate cause: an inability to assess the link between cool temperature and O-ring damage on earlier flights. Such a pre-launch analysis would have revealed that this flight was at considerable risk”. Tufte does not include the complete set of 13 charts used by Thiokol engineers the night before launch, but references Diane Vaughan’s The Challenger Launch Decision, where they are included on pages 293-299.
I have owned The Challenger Launch Decision for at least a decade, but never got beyond the Table of Contents. It’s one of those daunting volumes that you know you should read, but somehow it’s never the right week to take it on. I feared it would be dense and dry, but when I dived into Chapter 8, The Eve of the Launch Revisited, I became immersed in an utterly compelling drama: a reconstruction of presentations, questions, sidebars, interpretations and votes in that pivotal meeting, along with post-facto interpretations from key players. There is impatience and frustration and diverse opinions, but no indication of a lack of integrity from anyone involved. The final decision was devastating, given how things played out the next day, but no obvious malice was involved. Vaughan provides a masterful examination of the decision-making process, with all its human emotions, biases and diverse communication styles. She introduces the concept of “normalization of deviance”, which continues to influence NASA system reviews to this day.
What about the specific presentation on the temperature effect? The title of the presentation is “Temperature Concern on SRM Joints”. The final recommendation is that Launch should not proceed with O-ring temperature below 53 deg. Whatever other contributing factors for seal failure might have been discussed, this was a decision about temperature. Vaughan quotes Boisjoly as saying later “I then said that SRM-15 had much more blow-by indication and that it was indeed telling us that lower temperature was a factor.” That’s the essence of Tufte’s point: the presenters thought lower temperature was a factor, but nowhere in the charts is there a visual relationship between temperature and degree of damage.
As for Tufte, he may not have made ethical accusations, but there’s a strong case that he is not immune to re-working the facts of the case to fit a preferred narrative. While he is rigorous with guidance on visual story-telling, he could pay closer attention to the written and spoken work. Tone is important, and extravagant adjectives can be “speech junk” that distracts from the essential story much as “chart junk” can do. While we may not intend critique as ad hominem attack, we should be cautious about even the possibility of interpretation as personal denigration.
I also think that Tufte mischaracterizes the proximate cause. Where he observes “an inability to assess the link between cool temperature and O-ring damage on earlier flights” I might re-phrase it as “an inability to explain the link between cool temperature and possible booster joint failure”. I would personally choose a “failure progression scenario” framework to talk about the physics of failure, and isolate the steps where cold temperatures could contribute to a runaway problem. Tufte concentrates on flight experience, but the test data on secondary O-ring resiliency is also powerful. Recognizing that there are several contributing factors helps us appreciate why a temperature effect is “noisy”. I don’t think there’s a unique ideal chart that is guaranteed to make everything clear. Over-simplification tends to make good engineers suspicious. Data that is not “on trend” must not be ignored, and it should be explained.
As an aside, I’ll mention a caution from Tufte that I associate more with the Columbia disaster (and another of his books, Beautiful Evidence): reporting complex findings in Powerpoint is fraught with peril. The content is generally incomplete, intended to be supplemented with spoken presentation. Generally the file survives without the presenter’s explanatory exposition, and misinterpretation of incomplete information is all too easy.
Scientific American has a nice discussion of Tufte’s top-level ideas, summarized in the following 6 principles:
“(1) documenting the sources and characteristics of the data, (2) insistently enforcing appropriate comparisons, (3) demonstrating mechanisms of cause and effect, (4) expressing those mechanisms quantitatively, (5) recognizing the inherently multivariate nature of analytic problems, (6) inspecting and evaluating alternative explanations.” In brief, “information displays should be documentary, comparative, causal and explanatory, quantified, multivariate, exploratory, skeptical.”
He applies these principles to information displays, with a concentration on visual display. But they apply to the construction of engineering stories in all formats.
After this re-evaluation, Visual Explanations and Beautiful Evidence remain keepers. I’ll continue to pay attention to Tufte’s forceful exhortation to eliminate “chart junk” and concentrate on visual representation of causal reasoning. But I’ll think of Tufte when writing or speaking text that accompanies the graphics, and guard against “speech junk” too.
As a bonus from this investigation of information and misinformation in Challenger stories, I learned that Vaughan’s book has been a “sleeper keeper”. It emphasizes the human element in engineering and warrants more of my attention going forward.
I’ll continue to use the Challenger case study in discussions of risk perception, but with more context and humility.