Does the hockey stick “matter”?
http://sciencepolicy.colorado.edu/prometheus/archives/climate_change/000630does_the_hockey_stic.html
Post by Steve McIntryre and Ross McKitrick
Posted by Steve McIntyre
November 14, 2005
Stefan Rahmsdorf and others (including Roger Pielke, the proprietor of this site) have taken the position that the Hockey Stick is irrelevant to the great issue of the impact of 2xCO2 on global climate. Even the originator of the Hockey Stick, Michael Mann, who received many awards and honors for its construction, ironically has taken the position that it doesn’t “matter”. (I do not believe that he has not returned any of the honors.) I’m inclined to agree that, for the most part, the Hockey Stick does not matter to the great issue of the impact of 2xCO2. However, I believe that it matters (or should matter) to IPCC, to governments that relied on IPCC and to climate scientists who contributed to and supported IPCC and to people who may wish to rely on IPCC in the future.
The Hockey Stick was not, as sometimes portrayed, an incidental graphic, buried in IPCC TAR. Nor was it an icon resurrected by sceptics purely to torment poor Michael Mann. It could almost characterized as the logo for IPCC TAR. Figure 1 below shows Sir John Houghton, at the press conference releasing IPCC TAR, standing in front of the Hockey Stick. The graphic was used repeatedly in IPCC TAR and was one of the most prominent graphics in the Summary for Policymakers. Some governments (and, the Canadian government in particular) relied upon it in their promotion of Kyoto policy even more than IPCC. In the lead-up to adopting Kyoto policy, Canadians were told by their Minister of the Environment that “1998 was the warmest year of the millennium and 1990s the warmest decade”. So even if the Hockey Stick did not “matter” to the scientific case, it mattered to the promotion of the scientific case. Scientists may want to “move on”, but institutions cannot, if they want to maintain any credibility. If the Hockey Stick was wrong, it would be as embarrassing as the failure to find WMD in Iraq. In both cases, the policy might well be justified on alternative grounds, but the existence of the alternative grounds does not mean that responsible agencies should not try to isolate the causes of intelligence failure and try to avoid similar failures in the future.
The issues surrounding the MBH Hockey Stick are complicated by IPCC TAR statements and decisions, which, in retrospect, seem misguided, although there is little to suggest that IPCC AR4 is taking to steps to avoid similar potential problems. The most questionable IPCC statement about the Hockey Stick is that the MBH98 reconstruction had “significant skill in independent cross-validation tests”. I added bold to highlight the plural—a second level to the misrepresentation contained in this claim. The statement appears to have been written by Michael Mann about his own work. It is now known that the MBH98 reconstruction in the controversial 15th century portion failed the majority of cross-validation tests, including the standard R2 test [McIntyre and McKitrick, 2005a]; the source code provided to the Barton Committee shows that the adverse cross-validation R2 statistics were calculated, but not reported. It is also now known that the MBH98 reconstruction does not live up to its warranty that it is robust to the presence/absence of all dendroclimatic indicators, as the reconstruction depends on the inclusion of bristlecones, a series known to be potentially contaminated as a temperature proxy. Again, this adverse information was known to the authors and not reported.
If I were in Houghton’s shoes, I would be mad as a boil about all this. Since Houghton has a sincere belief that the impact of 2xCO2 is the great issue of our times, then, if I were Houghton, I would be particularly angry at being placed in a position where I used this logo and wasn’t fully informed about adverse information pertaining to it. I also wouldn’t be leaving it up to some probably adversarial committee like the Barton Committee to sort this out. I’d be all over the problem so that my community, the community of climate scientists, was not further embarrassed and so that government institutions would be able to rely confidently on the opinions of IPCC.
If I were Houghton, one line of argument that I would not accept is that the other “independent” studies all say similar things. It was the Mann study that I stood in front of. If there are serious problems in it, which were known ahead of time and I didn’t know about them, I would carve everyone involved a new you-know-what. Now for public purposes, I’d feel a lot happier if I could at least retreat to the safe haven of other studies that showed something at least similar to the Mann study. But I’d be pretty worried about them on a couple of counts and I’d want them torn through from top to bottom. The first thing that would worry me is that the studies were not really “independent”. The coauthors all seem to swap places: you see Mann, Jones, Briffa, Bradley, Cook, Schweingruber – all well-known scientists, but all having coauthored together. I’d be worried about a monoculture and want a fresh set of eyes. The second thing that would worry me is that the same proxies are used over and over – the bristlecones, the Polar Urals etc. I’d be worried about systemic problems. I’d be worried that no one seemed to have gone through these other studies like M&M had gone through the MBH studies. Maybe there are more time-bombs. I wouldn’t just passively wait for them to go off.
If I were Houghton, I would be enraged at the public refusal by IPCC authors to show their data and methods. When I read in the Wall Street Journal that Mann had said that he would not be “intimidated” into showing his algorithm, I’d have taken immediate action; I’d have told Mann to stop acting like a prima donna, to archive every line of code and data used in MBH98 and stop fighting a pointless battle that simply embarrassed IPCC and the entire field of climate science. I’d have done more than that. I’d have notified everyone contributing to IPCC that we did not expect the same kind of nonsense any more, that anyone contributing to IPCC would have to ensure that their archives of data and methodology were complete or else we couldn’t use their articles. I’d have done so before I heard from some redneck Republicans.
I would also review how we were checking studies in IPCC AR4. If our very logo for IPCC TAR blew up on us, then something was wrong with our procedures for review. I wouldn’t go around patting ourselves on the back and telling everyone that this was the most “rigorous” review procedure in the history of science, since we’d goofed on such a prominent issue. I’d want to know why we goofed and how to avoid it in the future, or at least, how to minimize the chances of a recurrence. So when some redneck tried to use the Hockey Stick fiasco against IPCC, I’d at least have an answer.
A final thing that I’d ask myself: if this damn chart is “irrelevant” to the great issue of 2xCO2, why did we use it at all? And why did we rely on it so much in our sales presentations? Why didn’t we just talk about the issues that were important and stay away from little irrelevant stuff? Maybe I’d find out, when I investigated, that someone had decided that this was merely for sales promotion – the climate equivalent of a sexy girl sitting on a car. If that were the case, I wouldn’t necessarily be happy about it, but at least I’d understand it. Then I’d want to make sure that we were also selling steak as well as sizzle. I’d sure want to make sure that we’d really done a good job on the issue which Ramsdorff and others now say was the “real” issue: climate sensitivity to 2xCO2.
Here I’d be bothered by how little guidance we actually gave to policymakers interested in an intermediate-complexity analysis of whether 2xC02 will lead to a temperature increase of 0.6 deg C or 2.6 deg C or 5.6 deg C. When I re-examined the TAR, I’d notice that we’d virtually skipped over these matters. I’d think: it’s not enough just to list all the results of different models; let’s try to figure out why one model differs from another, what are the circumstances under which a model gives a low sensitivity and what are the circumstances that a model has high sensitivity – if that’s the “real issue”. When I saw that we’d barely touched this sort of analysis in IPCC TAR, I’d be pretty embarrassed. I would certainly vow that in AR4, we would not repeat the mistake of ignoring the “real issues” in favor of hood ornaments.
The other thing that I wouldn’t do is simply ignore the problem and hope that it goes away of its own accord. I wouldn’t rely on the assurances of Mann and similar protagonists that the various alleged defects do not “matter”. No corporation would do so in similar circumstances and IPCC shouldn’t either. I would long ago have got some independent statistician to see if there really was a problem that I should be worried about. I wouldn’t have stood still for this water torture. I’d tell Mann to co-operate with the investigator and request McIntyre to cooperate. I’d try to get the parties to sign off on an exact statement of points and issues that everyone agreed on and ones that were in dispute. Once I saw what was in dispute, I’d ask for what would be involved to determine once and for all who was right on specific issues. I would long ago have gotten tired of barrages from both sides, where I couldn’t be sure that they were not at cross-purposes.
So does the Hockey Stick matter? Yes, if you’re a climate scientist that believes that the IPCC is an important institution whose opinions should be valued. Mann now thinks that the Hockey Stick does not matter. As so often, life is full of ironies.
Posted on November 14, 2005 05:57 AM
*****
Post by Ross McKitrick
Roger Pielke Jr. has posed a challenge to Michael Mann and us to briefly explain why each of us thinks the ongoing hockey stick debate matters. The technical content of the debate is summarized elsewhere ( HYPERLINK "http://www.climateaudit.org" here and HYPERLINK "http://" here; Our papers are linked under the heading “articles” (right hand column), and an overview paper by Ross McKitrick is HYPERLINK "http://www.uoguelph.ca/%7Ermckitri/research/McKitrick-hockeystick.pdf" here) and I won’t re-cap it here. That it matters is demonstrated by the enormous traffic on blog sites, the volume of comments to science journals, the opening of a Congressional investigation, etc. Obviously a lot of people find that it matters.
So: why does it matter?
1. It matters because it concerns the validity of an influential scientific paper. Mann’s 1998 and 1999 papers (which I’ll call “MBH”) have been heavily cited and highly influential. The paleoclimate field seems to have organized itself around them: other papers since then have gained prominence in proportion as they appear to back up MBH, whereas papers that contradict it have little prospect of being published or are relegated to lower-profile outlets. A popular icon in paleoclimate circles these days is what can be called a “spaghetti graph,” showing a pastiche of climate reconstructions from a small group of authors who call themselves the “Hockey Team”. They agree on few details, other than that the Medieval Warm Period is not as warm as the 20th Century.
Yet MBH turns out to have major flaws that fundamentally undermine its conclusions. These issues are interesting in their own right because MBH is a famous paper. But they also have wider scientific implications. MBH was “helped” along to its conclusions by some very convenient decisions about small changes to methodology, small edits to data series and not-so-small decisions about using contaminated bristlecone data. Maybe some of the other studies that appear to confirm MBH were also “helped” along so they would appear to agree with it. Efforts to evaluate the whole spaghetti graph has encountered maddening secrecy by the other authors concerning their data and methods, just as with MBH. But enough has been discovered to support a couple of assertions.
(a) The other spaghetti graph diagrams lack robustness. They all depend on delicate editing of weak data and just-so methodology. None are al dente: these are very soft noodles, and a plateful of weak results does not add up to a strong conclusion.
(b) There is an unexamined problem of spurious statistics in multiproxy constructions. Hockey team methods mine autocorrelated proxy data for simple correlations with autocorrelated temperature data. It is a classic recipe for spurious results, as has long been known in econometrics following the seminar studies of Granger, Engle and Phillips. What was predictable on theoretical grounds is now emerging empirically: proxies that extend past 1980 have no explanatory power for recent temperatures. And by implication, the existing corpus of multiproxy studies provides spurious information about the historical climate. Despite occasional claims of technical rigour, none of the spaghetti graph lines come from papers where the spurious regression problem was dealt with.
2. It matters because it exposes the uncomfortable reality about journal peer review. MBH(98) was published in Nature, considered by some the world’s “leading” scientific journal. Nature never verified that data were correctly listed: as it happens they weren’t. Nature never verified that data archiving rules were followed: they weren’t. Nature never verified that methods were accurately stated: they weren’t. Nature never verified that stated methods yield the stated results: they don’t. Nature undertook only minimal corrections to its publication record after notification of these things, and even allowed authors to falsely claim that their omissions on these things didn’t affect their published results.
In light of this, it is far past time for a wide-ranging discussion on what ‘peer review’ actually is. Policymakers routinely appeal to it as some kind of quality assurance guarantee. But obviously it isn’t. It serves some purpose internal to the world of scientific publishing, but policymakers’ beliefs about what peer review guarantees are for the most part sheer fantasy.
3. It matters because it exposes the uncomfortable reality about the IPCC. The IPCC’s use of the hockey stick was not incidental: it is prominent throughout the 2001 report. Yet they did not subject it to any independent checking: revealing an astonishingly cavalier attitude to the quality of their case. This raises the question of whether anything in the report was subject to serious, independent checking. They allowed chapter authors to heavily promote their own work with little or no oversight. They published false claims about the hockey stick’s statistical robustness and have never made any effort to retract them. On the basis of the MBH claims, their 2001 report reversed their 1990 conclusions about the MWP, and over-rode their 1995 warnings about not relying on bristlecone data, in order to promote the conclusions implied by the hockey stick. They encouraged governments around the world to rely heavily on a graph they themselves had not independently checked. One reason the hockey stick debate matters is because it exposes as worthless the guarantees given up to now about why the world should rely on the IPCC.
In this light I have no patience for the reaction by scientists to the Barton investigation. Why shouldn’t legislators begin asking questions about how the IPCC (and its allies in the science community) produce their reports? Policymakers have strong evidence that the IPCC process did not actually involve the rigorous checks and balances that they boasted of when releasing their 2001 report. It would be negligent of lawmakers not to open a wide-ranging investigation of this. Anyone who thinks the Barton investigation is unnecessary must think that IPCC reports don’t really matter: but they do, which is one reason why the hockey stick debate also does.
4. It matters because it exposes the uncomfortable reality about how governments use scientific information Canada (and many other countries) used the hockey stick heavily in their promotion of the Kyoto Accord. It is still prominent in government publications. Canada boasts of having spent $3 billion on climate change initiatives, much of it going to research. Yet for all the billions of dollars spent, and for all the proliferation of staff working on the matter, no one in government checked the hockey stick. Even when Canada’s chief climate science advisor and the Prime Minister’s own scientific advisor were personally informed about flaws in the hockey stick, no effort was made to remedy the government’s error. We have never been contacted by a single federal government scientist or other staff member for information on this topic, even though Environment Canada has in the past made heavy use of the hockey stick and more recently has issued communications supposedly providing “expert” commentary on my work, commentary that is predictably fallacious. Governments apparently use science when it suits them, as a promotional policy tool, with little regard to the facts of the matter. Perhaps it is naïve of me to have expected otherwise, but the realization still disappoints.
5. It matters because it exposes an uncomfortable reality about the culture of climate science. It took two outsiders to do all this work. Climate scientists in the field ignored the glaring problems in MBH for five years, and only seemed to get engaged after Stephen McIntyre and I began publishing our work. Since then the “engagement” of climate scientists has primarily consisted of ridicule, nitpicking, obstruction and catcalls from prominent scientists, especially those involved with the IPCC and the University Corporation for Atmospheric Research on Colorado. The few who have offered support tend to do so privately or anonymously.
We have tried to get data and methods from a long list of IPCC scientists who published prominent papers. The near-universal response is a hostile refusal to disclose, followed by stonewalling, delay and excuses. Lately a few have taken to publishing gripes and grievances about me in Eos. They complain that they don’t have time to archive their data sets, yet they seem to have lots of time to write editorials complaining about my inquiries.
These climate scientists seem to want to have things both ways. They are demanding that society set costly policy based on their work, yet they refuse to allow scrutiny of it. As a resident alien in IPCC-land, I have found it to be a culture of secrecy and conformity, to a degree that is incompatible with a healthy, vigorous intellectual culture. They can’t escape external investigation forever, and when it happens I believe a lot more skeletons will fall out of the closets.
And I do not believe I am alone in drawing such conclusions. The opening of the Barton investigation is the tip of the iceberg. The big scientific organizations that hyperventilated about it failed to note the ridiculous contradiction in their position. They insist that the scientific community should be left alone to handle the task of reviewing and critiquing influential studies, yet they not only failed to do it when it was needed, but routinely acquiesce in the widespread culture of secrecy that effectively prevents it from happening. It was only a matter of time before these issues got put on the table and subject to a top-to-bottom examination. The hockey stick debate seems to have been a catalyst, one more reason it matters to so many.
Posted on November 14, 2005 06:03 AM