Climate Models: An Assessment of Strengths and Limitations
Comments on the Draft Prospectus for Synthesis and Assessment Product 3.1

Reviewers

André GABUS
E-mail:

Wendy Gordon, Ph.D.
Texas Parks and Wildlife Department
3000 S IH 35, Suite 320, Austin, TX 78704
Phone: 512-912-7034
E-mail:
Area of expertise: Ecohydrology

David Karoly
School of Meteorology, University of Oklahoma
100 E Boyd St., Norman OK 73019
Phone: 405-325 6446
Fax: 405-325 7689
Email:
Expertise: evaluation of climate model simulations using observational data, climate change detection and attribution

Haroon S. Kheshgi
ExxonMobil Research and Engineering Company
Route 22 East
Annandale, NJ   08801
Phone: +1-908-730-2531
FAX: +1-908-730-3301
E-mail:

Robert E. Livezey
NOAA/NWS
W/OS4, Climate Services Division
Rm. 13348, SSMC2
1325 East West Highway
Silver Spring, MD 20910
Phone: 301-713-1970 X182
E-mail:
Expertise:  Intraseasonal to Interannual Modes of Climate Variability and their Prediction and Impacts,  Model Validation and Verification, User Services

Michael MacCracken
Climate Institute
1785 Massachusetts Ave., NW, Suite 100, Washington DC 20036
Phone: 301-564-4255
E-mail:
Areas of expertise: Climate change, climate impacts, and assessment

William F. O’Keefe
George C. Marshall Institute
1625 K Street, NW
Suite 1050
Washington, DC 20006
Phone:    (202) 296-9655
Fax:  (202) 296-9714
E-mail:
Area of Expertise:         Climate change policy

Name: Bruce A. Wielicki
Organization: NASA Langley Research Center
Mailing Address: Mail Stop 420, Hampton, VA 23681
Phone: 757-864-5683
E-mail:
Area of Expertise: climate data and comparisons to climate models

General Comments

Wendy Gordon, Texas Parks and Wildlife Dept.

If the intended audience includes researchers in non-climatic disciplines, please be sure that both hydrology and ecology are represented in the work that gets done.  None of the lead authors appear to represents these disciplines.  It is really important to try to get a better handle on the tradeoffs that might occur under climate change scenarios with respect to the water cycle.  Will ET increase to such an extent that there will be noticeable reductions in streamflow or groundwater recharge?  Will increases in precipitation result in increases in net primary productivity such that streamflow will remain unchanged?  There are lots of possible outcomes involving vegetation and the water cycle and it is important to start pinning down these joint processes under the wide range of climatic conditions and ecoregions that exist.

David Karoly, University of Oklahoma

General comments: Lines 22-24 state that “this CCSP product will focus on natural and human-caused factors influencing climate variability and change during the period 1870-2000”. However, none of the questions appear to specifically address the role of human-caused factors in influencing climate variability and change over this period, nor the relative importance of human-caused compared with natural factors in explaining observed climate change over this period. While question 4 asks how well do climate models simulate natural variability, none of the questions ask how much of the observed change at global or regional scales is likely to be due to the response to human-caused factors. 

The performance of climate models in simulating the climate variations over the 20th century is critical in evaluating their limitations.  Hence, I recommend the inclusion of an additional question “How well do climate models simulate the global and regional scale climate variability and change during the period 1870-2004 and what are the likely relative contributions of natural variability and human-caused factors?”.

While this might possibly be included within the current question 5, the text elaborating on question 5 does not mention the possible role of human-caused factors in possibly contributing to regional climate change.

In addition, why is the time period restricted to 2000 in this analysis and not extended to the most recent observational data, say up to 2004?

Haroon Kheshgi, ExxonMobil Research

An assessment of the abilities of climate models used in global change is a welcome focus.  However, the prospectus does not seem to address the difficult questions regarding model validation, model structure uncertainty, or what may be arbitrary, but commonly assumed, modeling choices.  Instead the prospectus seems to fall back on the well-worn ground of model simulation and the evaluation of these simulations.  While simulation/evaluation is an important contributor, its consideration is not sufficient to define uncertainty or the limitations of climate models and is apt to give a false sense of certainty.  Therefore, I suggest that this prospectus be broadened to consider all areas required to define model uncertainty and limitations.  Specific suggestions are given to broaden the prospectus questions.

Robert Livezey, NOAA

1.In my view the intended audiences and uses of this CCSP product (page 1, lines 30-46) are only addressed in questions 3-5 (page 2, lines 20-40), and most directly and relevantly in 4 and 5.  As posed the other questions entail either expositions of climate processes and their modeling or speculations about future directions in modeling, both of little interest to the intended audiences and distracting to them.

2. None of the lead authors are prominent experts in knowledge areas needed to authoritatively address questions 4 and 5.   

Michael MacCracken, Climate Institute

First General Comment: There is an overall tone to this prospectus that the climate modeling community (as represented by this particular set of authors) is in command of how results from climate model results can and cannot be used. This is an in appropriate tone to convey. Each potential user of the model results will have their own needs and constraints, and no general rule can be applied to what is or is not appropriate, especially given the broad set of variables, time and space dimensions, potential uses, different approaches to adjusting and using model results, and varying perspectives on how much information and confidence in the results is needed before considering various types of response or action. Instead, this product should present, as it does indicate it will do in some, but not all, places, the “strengths and limitations” of the model results. It is then up to the potential user to act responsibly in use of the results that are provided.

Second General Comment: The range of expertise of the author team is limited almost exclusively to the performance of general circulation models. There is no one among the set of proposed authors with any substantial experience on the application of global model results for any types of environmental or societal impact studies. There is not even a member of the lead author team who is a specialist in analyzing the model results for the types of information that those who might want to use the model results are most interested in, as for example in analyzing the model results for the relative ability to simulate extreme events, various types of climate oscillations, and so on. This just makes even more clear that the authors of this report should stick to indicating the strengths and weaknesses of model performance and not making inferences nor drawing conclusions about how the model results should or should not be used.

Third General Comment: The set of lead authors does not include anyone who has been actually working in using model results on a regional scale (e.g., on the subcontinental scale that the IPCC refers to as regional) nor anyone involved in downscaling model results for use in particular regions in particular ways, including recalibrating them to account for systematic biases, using empirical based techniques, and using the results to drive regional models on various scales. That the call for this report indicates that such regional methods are to be evaluated would seem to indicate that an expert in such approaches should be included on the team, especially given the biases that global modelers sometimes have against these techniques (offering in lieu of using such techniques only the unacceptable option of waiting until global models have adequate resolution). One or more experts in regional model validation and in regional downscaling techniques should be added to the author team and it should be made more clear that an unbiased analysis of strengths and weaknesses of such efforts will be carried out (though, in reality, it might be better for that analysis to be an entirely separate report, as is done by IPCC).

Fourth General Comment: The focus of the analysis is almost exclusively on the behavior of the atmospheric component of the climate models, and not on the coupled performance of the atmosphere-ocean-ice-land system, and primarily to the physical rather than the chemical or biogeochemical aspects. If, indeed, an analysis of all of the aspects of models is needed, the scope and set of authors needs to be greatly expanded. If instead the focus is going to be mainly on the behavior of the atmospheric component of the climate models, this should be openly said and perhaps the title of the product should be changed.

William O’Keefe, George C. Marshall Institute

This prospectus assumes that climate models are valid representations of the climate system, when, in fact, no climate model has been validated.  Validation of a model requires that it be developed on one set of data then shown to accurately represent a second, independent set of data.  This lack of validation is a critical issue for the intended audience for this study: “… decisionmakers and researchers who use climate model output as input to studies or analyses in their respective, non-climatic disciplines …”  Yet at no point does the prospectus address this issue. 

Several models have been calibrated to provide a reasonable match for the past climate of the past 130 years.  However, this calibration procedure does not demonstrate that these models can or will provide accurate projections of future climate.  While a separate product (Product 3.2) is planned to deal with climate projections, it is clear that the intended audience for this product, who are trying to assess the impacts of potential climate change, are more interested in those projections that in analyses of past climate data.  The current design of Product 3.1 will leave the incorrect impression that the limitations and uncertainties in climate models are understood, when, in fact, they have never been fully addressed.  Only through a rigorous validation processes can these limitations and uncertainties be fully quantified.  A fuller discussion of the issues involved in model evaluation is presented in the Marshall Institute’s report: Climate Models: A Primer (O’Keefe and Kueter, 2004).

We urge the CCSP to redesign this project to include a full discussion of the challenges involved in the validation of climate models and the implications of using unvalidated models for projections of climate change impacts.        

References

O’Keefe, W. and J. Kueter, 2004: Climate models: A Primer. George C. Marshall Institute.  

Bruce Wielicki, NASA Langley Research Center

I agree with the structure of the assessment report, and with the questions asked by the report.  The content is reasonable, and the set of co-authors is appropriate to the task.

Specific Comments

Michael MacCracken, Climate Institute

Page 1, line 15: The word “predicting” should be changed to “projecting” so as to really be clear that there has never been an intent to actually predict climate change in any detail out 100 years and more into the future.

Michael MacCracken, Climate Institute

Page 1, line 20: The word “topic” should be changed to “topics” (and “is” changed to “are”). Throughout this prospectus, this statement about the subject of this product should be used, including especially the terms “strengths and limitations.” Switching over to the term “uncertainty and limitations” should be avoided, especially as there is no real way to objectively define what the “uncertainty” is due to limitations in observations, that we only have one manifestation of the Earth’s climatic course, and because there are processes not included in the models.

Michael MacCracken, Climate Institute

Page 1, line 21: The phrase “guidance for appropriate use” is totally inappropriate. It is presumptuous and inappropriate for the climate modelers and for the CCSP/NSTC leadership who ultimately approve this report to claim that they alone can set the guidance for how model results can be used—how results are responsibly used depends on the particular situation and question, its context, and the information needs of the decision-makers. This product should, as the first sentence of this paragraph indicates, provide information on the strengths and limitations of the results from climate models, doing so in ways that will allow the potential user of the information to evaluate how best it may be applied or not applied.

Haroon Kheshgi, ExxonMobil Research & Engineering Company

Page 1, line 24: The intended users listed in the prospectus are those who use climate model output presumably for impacts studies which would presumable be for future climate change.  But the 2nd paragraph states that the focus of this assessment will be on past climate change.  Apparently, the goal is to consider uncertainties in future forecasts based on the simulation of the past along with well-known uncertainties and gaps in understanding of the climate system.  Suggest that this be made clear by changing line 24 to read "2000, and will seek to characterize all sources of uncertainties in climate models and their implications on the uses and limitations of climate model results for future climate change."

Michael MacCracken, Climate Institute

Page 1, line 32: “biogeochemical cycling” is another example of an important use, and should be included.

Michael MacCracken, Climate Institute

Page 1, line 34: “uncertainty and limitations” should be changed to “strengths and limitations”—it is impossible to determine the importance of limitations without have a sense of how well models are performing.

Michael MacCracken, Climate Institute

Page 1, line 36: The term “regional” needs to be defined. IPCC considers regions to be areas as large as Asia, whereas others might call the San Francisco Bay Area a region. It is indeed true that users want information on finer than global scales, but this can cover a very wide range, and it will be important to differentiae comments about geographical patterns of the changes by scale. The analysis of the strengths and limitations of the regional patterns of results also needs to make very clear that what is really at issue is to what degree the models can indicate the departure of a local to regional result from the larger average value, and not whether there will be some specific change at a particular location (i.e., if there is good agreement that the world will warm 2 to 4 C, then what is important is whether regions will warm more or less than this average; too often, however, an indication that regional patterns cannot be relied upon is taken to mean that then there will be no change at that location). Already there is some regional guidance, namely that land will warm more than the global average value and oceans less, high latitudes will warm more than the global average and low latitudes less, that the snowline will generally rise in ways that tend to reduce snowpack, etc.

Michael MacCracken, Climate Institute

Page 1, lines 36-37: Dynamical downscaling is only one of a number of (complementary) techniques that are used to develop scenarios for what types of changes can be expected on the regional scale. In that there is such a broader set of approaches, it would seem more appropriate to consider all such approaches, and in fact the use of combinations of approaches that can each help limit the problems with other approaches, in a separate assessment product rather than to do an incomplete job in this product.

Michael MacCracken, Climate Institute

Page 1, line 40: “uncertainty and limitations” should be changed to “strengths and limitations”—it is impossible to determine the importance of limitations without have a sense of how well models are performing.

Michael MacCracken, Climate Institute

Page 1, lines 41-42: It is, as noted in a general comment, presumptuous of the modelers to say what appropriate and inappropriate uses of model output will be as such a discussion necessarily involves value judgments regarding the application and nature of the situation. Including a discussion of the strengths and limitations of applying models at particular scales or for particular variables would be acceptable, as long as there is no implicit standard of performance inferred that must absolutely be met. This is particularly the case as quite often model results are not used directly but are adjusted in various ways to account for, for example, systematic biases, geographic features not represented in the models, etc.

André Gabus, Switzerland

Page 2, Line : Proposed Addition at end of paragraph 1: Their explaining strength will be evaluated (in the comparison of the models to be surveyed, those with a high content in causally-linked constituents will be ranked higher than those with only statistically-correlated factors).

Justification:  Information on sensitivity, feedbacks and uncertainties (focus of Product 3.1) is insufficient to convey confidence to decisionmakers for using climate model outputs as input to studies or analyses in their respective, non-climatic disciplines (e.g., ecosystem science, hydrology and water resources, economics, human health, and agriculture/forestry). Before to deal with model projections (next Product 3.2), an assessment of models' qualitative structure is needed. In addition to ability of models to reproduce past events and trends, their explaining strength requires to be evaluated for conferring confidence in their predictive value. Preference will be given to those with a high content in causally-linked constituents versus those with purely statistically-correlated factors. Qualitative structural analysis is a tool available for such an evaluation of models.

References

1. Fontaine P., M. Garbely and M. Gilli (1991), «Qualitative Solvability in Economic Models» in Computational Economics, 4: 285-301, Kluwer Academic Publishers.
2. Fontela E. and A. Gabus: Current Perceptions of the World Problematique[PDF], in C.W. Churchman et R.O. Mason (Eds), World Modeling : A Dialogue , North Holland/Elsevier, Amsterdam/New York, 1976.
   
3. Fontela E. and A. Gabus:  The DEMATEL Observer, DEMATEL 1976 Report, BATTELLE Institute, Geneva Research Center, 1976.
   
4. Gilli M. (1984), «CAUSOR – A Program for the Analysis of Recursive and Interdependent Causal Struc-tures» in Cahiers du Département d'Econométrie, Université de Genève, Cahier 84. 03, April.
   
5. Gilli M. (1992), «Causal Ordering and Beyond» in Intenational Economic Review, Vol. 33, N° 4, November,
   
6. Warfield, J.N. (1982). «Interpretive Structural Modeling». In S.A. Olsen (Ed.), Group planning and problem-solving methods in engineering (pp. 155-201). New York: Wiley.
   
7. Warfield, J.N. (1994). A science of generic design: Managing complexity through systems design. Ames, IA: Iowa State University Press.

Haroon Kheshgi, ExxonMobil Research & Engineering Company

Page 2, Line 3, question 1: Assessment of how models represent processes would add critical information in responding to this question (no underlying guidance is given in the prospectus).  Clarification of the types of parameterizations used for, e.g., clouds and other well-known sources of uncertainty, and the extent that various models span all conceivable behaviors would help characterize the contribution of model structure uncertainty in forecasts.  Suggest adding on line 8: "This section will survey the ways key processes are modeled, consider the validation of modeled processes, and identify gaps in understanding and poorly characterized sources of uncertainty. "

Michael MacCracken, Climate Institute

Page 2, lines 5-7: It is not clear from the write-up if the effects of using alternative formulations will be described (e.g., of convection). This point should be clarified.

Michael MacCracken, Climate Institute

Page 2, line 8: The use of the word “crucial” is problematic as it quite likely depends on the potential nature of the use of the model results (i.e., requires a statement of crucial to what?). It is fine to discuss how the inclusion or exclusion of a process may affect the strengths and limitations of model results, but whether a process (or forcing) that is left out is crucial is not, given the state of model development, something that can be evaluated without providing context. A point that will also need to be evaluated is whether the omission of a process affects the response of the model or just its base state, and so whether the process is important for limiting the range of estimates of model sensitivity. Also, recall that early weather models left out all radiation processes, yet still got some useful results—the report should list what the effect is of leaving something out, even if the answer is estimated, but should not be deciding what is crucial in any general sense. Note also that, at the end of line 8, the wording needs to be adjusted to say “climate and that are not yet”.

Haroon Kheshgi, ExxonMobil Research & Engineering Company

Page 2, Line 10, question 2: Given that the greatest source of uncertainty in recent radiative forcing change may well be from aerosols, a full account of aerosol forcing uncertainties is essential to this prospectus.  Suggest changing Lines 18-19 to state: "While Sythesis and Assessment Product 2.3 will focus on aerosols and their relationships to climate change, a full treatment of aerosols will be considered in this report so far as they affect the characterization of climate model uncertainty."  If this report is going to consider any recent climate trends in the estimation of climate model uncertainty, it is essential to factor in all relevant effects of aerosols.

Michael MacCracken, Climate Institute

Page 2, lines 13-15: It is not clear from the text whether the strengths and weaknesses of the various estimates of the forcings and how they are represented will be included. For example, variations in the Sun’s intensity have often been represented by altering all wavelengths by an equal percentage rather than disproportionately affecting the ultraviolet; little mention is ever made of the changing height of SO2 and aerosol emissions over time, even though such changes in altitude would have dramatically affected the atmospheric lifetime of the sulfate and aerosols during the 20th century. In that there will presumably be comparisons of model results to observations, the effects of uncertainties in the estimates of the forcings needs to be mentioned, and if possible, separated out.

Michael MacCracken, Climate Institute

Page 2, line 20: The opening question is one-sided and not really answerable (we simply do not have enough observations to make a full quantitative answer), and is stated in a one sided way. The question should be rewritten to ask, “How well do climate models simulate the observed climate, and to what extent are the differences within or outside bounds derived from observations and other techniques?” The second question should also be rephrased as it cannot be answered because uncertainty cannot be defined; an alternative phrasing would be: “In what ways has the ability to simulate the observed climate changed over time, and what has led to these changes?” If indeed the skill of models to predict changed over time, then this must presumably be derived from performance at simulating the weather or perhaps seasonal fluctuations (or perhaps paleoclimatic changes—which unfortunately seem not to be considered as a test case--although there are many uncertainties introduced by factors other than the model). The real difficulty here is that we really have very few situations on which to evaluate the ability of models to simulate decadal centennial scale climatic change, which is of most interest—and the limitations arising from this need to be indicated (as well as their implications for what can be expected from model validations studies).

Haroon Kheshgi, ExxonMobil Research & Engineering Company

Page 2, Line 20, question 3: The question recognizes the importance of uncertainty in defining a forecast of climate.  However, we do not know uncertainty, we can only estimate it.  Estimates of uncertainty commonly increase due to neglected sources of uncertainty, or because such estimates are contingent on a set of assumptions (partial estimates of uncertainty).  And estimates of uncertainty can be driven by factors in addition to knowledge.  Suggest 2nd sentence say "In what ways have estimates of uncertainty in model-based predictions increased and decreased over time, and what were the causes of these changes?".  Suggest changing line23-24 to read"…the identification of key model assumptions and how they may be a source of uncertainty, as well as the characterization of radiative forcing."  Suggest adding to line 25 "…knowledge or changes of key assumptions or uncertainty estimation methodologies can lead to greater estimated uncertainty, e.g., by increasing …"

Michael MacCracken, Climate Institute

Page 2, lines 22-24: The discussion should also consider the relative ability of models to simulate various integral measures of the climate in addition to simulation of particular variables.

Michael MacCracken, Climate Institute

Page 2, lines 28-29: Given the strengths and limitations about what we know, the wording should be changed to “and how has variability changed over time?”

Haroon Kheshgi, ExxonMobil Research & Engineering Company

Page 2, question 4: The ability of models to simulate variability does not imply the ability of models to predict changes of the statistics of variability.  Suggest the title of this question be changed to "How well can climate models predict natural variability and its change over time?" and change the term "simulate" in the underlying text to "predict".  This question would then require assessment of the validation of modeled mechanisms, which may make use of model evaluations (how well observed variability is simulated) but also would consider if alternate mechanisms, parameterizations, or other model assumptions could also simulate variability yet give different projections of variability change.  Limiting the scope of this question to only simulation and evaluation is not a sufficient basis for the uses of climate models and could potentially give a false sense of predictive capability.

Michael MacCracken, Climate Institute

Page 2, line 30: Again, it is inappropriate to use the word “crucial” as it is not the case in all situations (e.g., not necessarily the case for using model results to estimate the long-term rate of rise of sea level due to thermal expansion, etc.). It would be acceptable to say that the ability of models to do this is “of great interest to many in the impacts and applications community.”

Michael MacCracken, Climate Institute

Page 2, line 31: It would be helpful to change “these” to “relevant”

Michael MacCracken, Climate Institute

Page 2, lines 32-35: It is a bit surprising that model simulations of paleoclimatic change is not mentioned here. It is such larger and longer lasting changes that really are of most relevance for how well models are representing climate sensitivity. Also, it needs to be made very clear in addressing the relative skill to simulate these oscillations that there should be no expectation of a perfect match to particular events (in contrast to the test applied by Patrick Michaels, which in effect tested the ability of models to make a 100-150 forecast of specific events!!); rather, the issue is how well the models represent the variability in a statistical sense, and in doing this evaluation, it needs to be mentioned that we have only one manifestation of Earth history and that this record is complicated by both human and other natural influences, the uncertainty of each of which has an influence that will be hard to extract out.

Michael MacCracken, Climate Institute

Page 2, lines 38-39: Change “limitations” to “strengths and limitations;” again, one can evaluate the importance of limitations without understanding the strengths. On page 1, lines 36-37, this evaluation was limited to “dynamical downscaling,” but here the indication is that the evaluation will consider multiple downscaling methodologies; if this is the case, it is again particularly important to expand the set of authors and to more fully indicate what the evaluation will encompass. Also, it is important to recognize that there are various downscaling methodologies and that various uses are made of the results and various adjustments are made to avoid or limit systematic errors and various biases. Because of the various methods, applications, and adjustments that are used to generate plausible scenarios, there should be considerable reticence in drawing general conclusions (i.e., do not assume that impact analysts are not cognizant of various limitations and are not accounting for them, and do not assume that the presence of uncertainties necessarily invalidates conclusions). Again, determining whether there is a problem that is systemic (so affecting the base state) or more general (so also affecting the model sensitivity) needs to be differentiated.

Michael MacCracken, Climate Institute

Page 3, line 4: In these discussions, it will also be important to recognize the limitations in the observations, including in terms of limits due to problems of spatial and temporal resolution (e.g., for precipitation), that the observational history is only one manifestation of the system’s temporal behavior, and that the observational record is not from a system in equilibrium.

Michael MacCracken, Climate Institute

Page 3, lines 5- 7: Is it really meant here that “sea level” is going to be one of the key variables, or should this say “sea level pressure”? In any case, this is a rather limited set of variables to use to evaluate climate model performance, there being nothing relating to the state of the cryosphere, land surface, or oceans (unless sea level is really a proxy for everything happening in the ocean, the cryosphere, and for soil moisture—which would be a much too integrated measure). It is also not clear if just the mean values or the variability will be considered, including the seasonal range, storminess, etc. Also, the set of decision-makers who wants only this information is fairly limited—certainly anyone interested in impact analysis wants much more information.

Michael MacCracken, Climate Institute

Page 4, lines 6-10: This set of proposed authors, although including leaders in their field, has a fairly limited disciplinary range. For example, if one really wants to address the potential value of model results for impact studies, there should be impact analysts represented. To really evaluate the range of regional downscaling techniques, there should be regional experts. To really treat the range of oceanic, cryospheric, terrestrial, and biogeochemical behavior of the models, there should be experts in these fields. Actually, from the listing of experts, it very much appears that this is an evaluation of atmospheric rather than climate models.

Michael MacCracken, Climate Institute

Page 4, lines 12-13: If indeed the “lead agency in consultation with the supporting agencies” are the ones who are going to determine the set of authors, and if indeed the lead agency and other agencies are going to determine the final text through their control of the final review, revision, and publication process, then it very much appears that this group is being organized in a manner that is subject to the Federal Advisory Committee Act, and should therefore meet the requirements of FACA. Going through what will be a challenging effort to prepare this report and then possibly facing difficulties because FACA was not studiously followed would be a major waste of time and effort for the CCSP and the authors. This matter needs to be fully clarified by legal counsel of the various agencies.

Michael MacCracken, Climate Institute

Page 4, lines 26-27: Change “uncertainty and limitations” to “strengths and limitations” as one cannot have context for evaluating the importance of the limitations without knowing the strengths.

Michael MacCracken, Climate Institute

Page 5, lines 8-10: The wording here makes it seem as if all the authors are allowed to do is to cut and past out of peer-reviewed literature and are not able to do any analysis on their own, create any graphics about results that are not in the literature, etc. This language seems way too tight and deferential to peer-reviewed literature, which can also be incorrect (as, for example, when there is a low quality peer-review process or an article is published in a journal focused mainly on other subjects). Also, much of the detail concerning the model formulations that might be relevant for their consideration is not contained in the peer-reviewed literature, although it is publicly available, and not all results of models that are relevant might be published in the form needed (e.g., full seasonal patterns, etc.). In that this product will undergo its own review, the phrasing here should be modified by, for example, deleting the word “all” on line 8 and inserting the phrase “to the extent possible” after “writing”. Roughly 20 years ago the AMS tried to impose a strict policy requiring peer-review outside the originating organization and found it unworkable, for it excluded reports from the National Academy of Sciences, the Bureau of Standards, etc. Another problem with this formulation is that it generally excludes anything in a book and items in various proceedings volumes that report on more recent results. If indeed there is confidence in the review process, there is no need to have such a stringent policy on this.

Michael MacCracken, Climate Institute

Page 5, lines 29-30: The prospectus should indicate that the review comments will be posted publicly as will a listing of all of the expert reviewers, even if not associating the comments with the authors. This is an important step to add to the credibility of the overall effort.

Michael MacCracken, Climate Institute

Page 5, lines 29-30 and Page 5, lines 37-38: It is commendable that the authors are being given responsibility for deciding how the review comments should be addressed. In doing so, however, there should be a requirement that the authors indicate in a publicly available manner how they have dealt with the various substantive comments.

Michael MacCracken, Climate Institute

Page 5, lines 43-45: The set of additional comments should be compiled and made public and the responses of the authors should be indicated; a listing of the agencies that commented should also be provided, just as a list of expert reviewers is provided. It is important that the phrasing included here, namely “resolution by lead authors,” means exactly that—that is, that the authors determine how best to address the comment, whether by making a change or responding why they will not make the change. An important step in maintaining the integrity of the process will be to have the set of changes documented and made available to the public.

Michael MacCracken, Climate Institute

Page 6, lines 4-5: The CCSP Guidelines indicate that the CENR will be responsible for both the final review and approval (concurrence) of the report. In particular, they state that: “The National Science and Technology Council (NSTC) will be responsible for final review and approval of the synthesis and assessment products. Products not cleared by NSTC cannot be released as disseminations of the Federal government. Consistent with NSTC procedures, approvals will require written concurrence from all members of the NSTC’s Committee on Environment and Natural Resources (CENR). All comments generated through the NSTC review will be addressed by the CCSP Interagency Committee. The CENR membership includes senior officials representing the Executive Office of the President and the 15 federal agencies with significant responsibilities for environment and natural resources programs.” Two issues with this: (1) the OSTP Web site lists no members of CENR and indicates its leadership positions are vacant. It needs to be explained what offices and whom will really be doing this review; (2) Requiring that every member of CENR concur very much makes this a formal government report and gives an indication that this process should be covered by the Federal Data Quality Act and its various requirements. For the record, the approval process for the US National Assessment only required that the CENR representatives concur that the report process met the guidelines—agencies were not put in the position of having to endorse every word of the report, something that seems particularly inappropriate (actually reminiscent of the Roman Curia deciding the Earth is at the center of the Universe). This process needs to be rethought and FACA requirements need to be considered so that difficulties with the report’s completion and publication do not arise later.

Michael MacCracken, Climate Institute

Page 6, line 7: It should be made clear here, for the sake of overall credibility, that the lead authors have the final word. In fact, provision should be made to allow for dissents by any of the authors if they deem it appropriate.

Michael MacCracken, Climate Institute

Page 6, line 18: It is not at all clear why the choice is made to cover only the period 1870-2000. Many of the recent model runs start earlier than this date, even providing an indication of model results before forcing begins and so enabling an evaluation of model stability and drift. While the instrumental record may only being in 1870, surely the historical period begins earlier, and so do a number of the key forcings.

Michael MacCracken, Climate Institute

Page 6, line 38: With the process of this review of the Prospectus taking an extra 1-2 months, it seems apparent that the draft of this product will not be available in early October at the time of the next major public meeting convened by the CCSP. Nor, apparently, will these analyses be done in time to serve as input for the IPCC. This seems unfortunate, and suggests that perhaps the CCSP meeting should be delayed.

Michael MacCracken, Climate Institute

Page 7, lines 4, 19, and 33: In that “Dr.” is used to open the bios for Dr. Kiehl and Dr. Rind, this should be done for David Bade, Curt Covey, and Isaac Held, all of whom have PhD degrees.