Also available: PDF version of this file

Reviewers

Name: Dr. Mark Abbott
Title: Co-Chair, Board on Oceans and Atmosphere
Organization: National Association of State Universities and Land-Grant Colleges

Name:   Dr. Mathew Barlow
Title:      Staff Scientist
Organization:     Atmospheric and Environmental Research, Inc.
Area of Expertise:          Observational analysis of large-scale climate variability

Name: Lisa Dilling
Title: Project Scientist
Organization: Environmental and Societal Impacts Group (ESIG), National Center for Atmospheric Research (NCAR)

Name:   William L. Fang
Title:     Deputy General Counsel and Climate Issue Director
Organization:   Edison Electric Institute (EEI)
Area of Expertise:    Energy and environmental issues, particularly as they relate to global climate change
Note: Please see Edison Electric Institute cover letter at end of collation

George Marshall Institute

• Name:   William O’Keefe
  Title:   President
  Organization:   George Marshall Institute
  Area of Expertise:   policy development and analysis
• Name:   Leonard Bernstein
  Title:   Fellow
  Organization:   George Marshall Institute
  Area of Expertise:   policy development and analysis

Name:   Kim Knowlton
Title:   Doctoral Candidate, Graduate Research Assistant
Dept of Environmental Health Sciences
Mailman School of Public Health, Columbia University
Area of Expertise:   Human health impacts of climate change

Name: Michael MacCracken
Title (1): President
Organization (1): International Association of Meteorology and Atmospheric Sciences
Title (2): Chief Scientist for Climate Programs
Organization (2): Climate Institute
A rea of Expertise: Climate change and climate impacts, assessment and summarization of findings

Name: Susanne C. Moser, Ph.D.
Title: Research Scientist
Organization: Environmental & Societal Impacts Group
National Center for Atmospheric Research
Areas of expertise: climate change impacts on coastal areas, human dimensions of global change, science—policy interface, public participation/stakeholder involvement in assessment activities.

Name:   John Mutter
Title:   Deputy Director
Organization: The Earth Institute at Columbia University

Name:   Michael T. Neuman
Title:   Administrator
Organization:   Preserve Our Climate Coalition
Area of Expertise:   Water Resources Research

Name: Claire L. Parkinson
Title: Senior Scientist
Organization: NASA Goddard Space Flight Center
Area of Expertise: Satellite passive-microwave sensing of polar sea ice

Name:   Michael J. Prather
Title:      Professor
Org:      University of California, Irvine
Mail:      Dept of Earth System Science, UC Irvine
area of expertise:atmospheric chemistry, composition and climate; former editor-in-chief of GRL

Name:   Susan Solomon
Title:   Senior Scientist
Organization: NOAA Aeronomy Laboratory
Area of Expertise:   atmospheric chemistry and climate, and assessment

Name: Kevin E. Trenberth
Title: Head, Climate Analysis Section
Organization: NCAR
Area of expertise: climate dynamics, climate system observations, hydrological cycle, energy cycle, El Nino, climate change, structure of atmosphere, reanalyses

Name: Sophia Oliver/Cathy Johnson
Title: Scientific Policy Advisor
Organization:  Global Atmosphere Division, UK Department for Environment, Food and Rural Affairs
Area of expertise: Climate science and science related climate policy

Name: Derek Winstanley
Title: Chief
Organization: Illinois State Water Survey

General Comments

Abbott, Nat’l Assoc. of State Universities and Land-Grant Colleges

On behalf of the National Association of State Universities and Land-Grant Colleges (NASULGC), thank you for the opportunity to comment.   We commend the open procedures CCSP has followed in each phase of the planning process.   The pains taken to solicit public input and ensure accountability will maximize stakeholder support and scientific integrity.   We trust that a similar level of openness will be maintained in preparing the CCSP synthesis and assessment products.   These draft guidelines are a very positive step in that direction.  

We support the inclusion of rigorous peer review and public comment periods for each product’s draft prospectus and draft document.   NASULGC wishes to assist the CCSP in preparing high quality products in whatever ways possible.   The academic research community should be involved in every step of the process, both to provide scientific expertise and to help shield CCSP from perceptions that political bias will affect the process.

American Petroleum Institute, Feldman

The Draft Guidelines generally describe a reasoned approach to producing the identified products and ensuring their appropriate review.

Table 2-1 indicates that a large number of synthesis and assessment products will be developed and released within two years and an additional pulse of products will be released in the subsequent two years.   It is likely that many of these products will be out for public comment simultaneously.   Given the importance of the products, the valuable input that may be obtained through public review, and the potential volume of the material to be reviewed, it would be preferable to extend the public comment period to a minimum of 60 days.

Barlow, Atmospheric and Environmental Research, Inc.

COMMENT: If the draft prospectus were publicized more widely, more advantage could be taken of the climate community's expertise.   The only specified posting is on the CCSP website and Federal Register.   While no doubt the distribution will be wider, a targeted campaign to inform, e.g., AMS and AGU members would garner more and better comments and, hopefully, better engage the climate community.   Broad engagement of the community is crucial to the "scientific integrity and credibility" mentioned in the first paragraph.

COMMENT:   Who are the "interested parties" who select authors?   Are these the same people as the "CCSP Principals" (also undefined)?   This is an important role and it would be helpful to see it clearly defined.   If the interest parties include anyone with relevant expertise who views the document, then a broader dissemination than just the CCSP website and Federal Register would seem to be in order.   Clarity on this point is quite relevant to the "scientific integrity and credibility" mentioned in the first paragraph.

COMMENT:   The specified review process does not sound like "peer review" to me, in that the traditional review process usually involves a third party, the editor, making final adjudications between reviewers and authors whereas here, as far as I can tell, the final decision on reviewers comments is made by the (unspecified) CCSP Principals, who are not disinterested parties.   If the CCSP Principals include government employees, as appears to be the case, political pressures may make it difficult for them to provide an unbiased assessment of reviewer comments and author responses, potentially seriously undermining the credibility of the process.

Dilling, NCAR

One of the most important lessons that we have learned from previous assessments and syntheses is that the process is as important as the information in the formulation of a successful product. The GEA group at Harvard has summarized it thus: credibility, saliency and legitimacy all must be balanced in every assessment process in order that the assessment achieves its presumed target—of providing useful, timely and relevant scientific information to decision-makers. I believe that certain aspects of each of these features are missing from the current draft guidelines for producing CCSP Synthesis and Assessment Products.

Let me first address legitimacy. Legitimacy can be defined as the “perceived fairness of the assessment process.” In this case, it means involving in as fair a way as possible as many participants as possible who hold a stake in the information and how it is conveyed. This might mean scientists, federal and state agencies, businesses, non-governmental organizations, etc. I am concerned that the description of the development of a prospectus and author selection (under Phase I) does not adequately address this issue of legitimacy. The description of Phase I does state that the lead agencies for each product will involve the public and scientific community to define the prospectus. However, given the extreme time constraint that many agency folks are working under, this could devolve simply into agency preparation of a prospectus and then 30-day circulation and peer review of a draft for public comment as required in line 37. This is not an adequate process for involving stakeholders. The questions, potential authors, overall goals etc. must be jointly defined with the groups most heavily affected by the information for the documents to be seen as legitimate. Setting the agenda and the questions is generally seen as the most important phase of the assessment process, and therefore the most critical phase for creating joint ownership by involving multiple stakeholders. A comment period is simply not interactive enough. Adequate time must also be allowed in order to build up the level of trust between participants in the process. The reason that the prospectus is so important is that the rest of the whole process and ultimate end product flows from that prospectus (page 2 line 9 and 10).

Credibility in this case will stem from the authority and position of the authors involved and the source of data. I am concerned that author selection remains only in the purview of the lead agencies (page 2, line 1). While agencies certainly are a major stakeholder in the assessment process, they are not the only stakeholders. Selection of authors is another key part of the process that should involve multiple stakeholders, so that the legitimacy will be enhanced. I suggest that authors be selected after an open process involving stakeholders from several groups. If the Federal Government is seen as the only “owner” of the assessment report process, the credibility of the product will be reduced.

Saliency is the third generally accepted criteria for producing a successful assessment process. Saliency is how relevant the information is to the audience. Again, this is an area that could be improved by making a stronger commitment to involving those outside the Federal government or scientific communities. If the questions are defined partly in the public, business, state and local government, and non-governmental organization realms, then the results of the assessment process will be much more relevant to them.

Second general comment: There are too many assessment reports promised in the CCSP.

Third general comment: The time for production of reports is very short, almost too short to do the process properly if you actually involve stakeholders. Reconsider which reports are really needed (maybe through an open stakeholder dialogue process??) and prioritize.

Fourth general comment: The role of Scientific Steering Groups (SSGs) is not mentioned. SSGs would be very helpful in representing the scientific community in these dialogues and processes.

Fifth general comment: The agencies are listed as the major entities responsible for the production of these assessments (page 4 line 18-22; page 4 line 26-38). It is not clear that the agencies have the time or mandate to play such a role. While responsibility certainly must be assigned, again, the CCSP might consider an alternate process that gives a wider range of stakeholders a role in these products, including of course the Federal agencies. Independent assessments, and processes that receive open debate and participation almost always have greater credibility.

Edison Electric Institute

According to the March 29, 2004, “Dear Colleague” letter on the U.S. Climate Change Science Program (CCSP) Website titled “Invitation to Submit Comments on Draft Guidelines for Producing CCSP Synthesis and Assessment Products,” the expected “products are described” in Chapter 2 of the July 2003 Strategic Plan and are “intended to provide useful information for a variety of end users about key climate change topics.”   The letter added, “The products include reports, data sets, and evaluations of the uses and limits of climate information in decision support.”

Chapter 2 of the Strategic Plan sets forth five CCSP “Goals” and provides in Table 2-1 a summary of 21 “Synthesis and Assessment Products -- Topics to be Covered” by 13 federal lead or supporting agencies over a period of four years starting October 2003 (according to the Website document).   According to the Strategic Plan, these agencies are: Agriculture, Commerce, Defense, Energy, Health and Human Services, Interior, State and Transportation, together with the Agency for International Development, Environmental Protection Agency, National Aeronautics and Space Administration, National Science Foundation,   and Smithsonian Institution.    As explained in Chapter 2 of the Strategic Plan, the table provides a “target time frame for completion of each product” that ranges from two to four years and indicates that products “will fulfill the requirements for updated synthesis and assessment” contained in section 106 (“Scientific Assessment”) of the 1990 Global Change Research Act.  

The Draft Guidelines appear to integrate, in part, some of the details set forth in the Principles and Procedures of the Intergovernmental Panel on Climate Change (IPCC) for the preparation, review, acceptance, adoption, approval and publication of IPCC reports with the requirements of applicable federal statutes, such as the 1990 Global Change Research Act, Federal Advisory Committee Act, Information Quality Act and those statutes specifically applicable to the lead and supporting agencies, such as the Department of Energy Organization Act of 1977, and their applicable regulations, guidelines and policies.   Indeed, Dr. Susan Solomon – a senior scientist of the National Oceanic and Atmospheric Administration (NOAA) and co-chair of IPCC Working Group I – gave a presentation about the draft guidelines in the context of the IPCC procedures at an April 8, 2004, “Workshop on Issues in Global Change” held by the Global Change Coordinating Committee at the National Academies.   She pointed out that the IPCC “takes more than 2 years to produce a major report” that involves developing a “zero-order draft, informal review, first order draft,” etc.; that the IPCC “assessment conclusions are fully grounded in well documented peer-reviewed literature”; and that this “IPCC system . . . means that the structure and content of the report is determined by scientists.”   She said there is “a clear demarcation of roles and procedures between the stakeholders/Governments and the scientists who produce the report,” which “gives credibility to the process and allows scientists to become involved without fear their work will be manipulated.”   In specifically addressing the guidelines, Dr. Solomon said that there is a need to “[c]larify and document exact roles of all involved in CCSP assessments and procedures to be followed, especially in the review process”; that the “CCSP review proposal does not appear to be staged to address its audiences sequentially”; that “rushed reports pay the price in quality”; and that the “CCSP would benefit from defining the process for stakeholder inputs.”

Since EEI participated in the development of the “Principles and Procedures Governing IPCC Work” over several years, culminating in their final adoption in April 1999 in San Jose, Costa Rica, we are quite familiar with them.   These draft guidelines appear to be an attempt selectively to borrow some portions of the IPCC provisions, which are well integrated in the IPCC process, while leaving aside many of the other parts of the IPCC provisions.   The “CCSP Guidelines” cannot and should not attempt to emulate portions of the IPCC procedures, which were developed after long debate primarily to address the roles and responsibilities of authors and governments of the IPCC in that process.   In short, the IPCC process is not well suited for these U.S.-only guidelines, which must comport with applicable federal law.

The underlying assessments are the work of the authors (i.e., scientists and technical experts).   They are reviewed by the appropriate IPCC Working Group and accepted by the IPCC, while the Synthesis Report is reviewed and approved by the governments acting as the intergovernmental panel.   The only opportunity for stakeholder input is through their governments, not directly at the IPCC sessions.  

Nevertheless, Dr. Solomon’s comments are generally on point with respect to the “roles” of all involved, including stakeholders from business, industry, agriculture, labor, environment, the public, government and non-government scientists, and other technical experts.   Their roles, the timing of their roles, and input opportunities need to be better clarified in the guidelines, taking into account the several statutes and related statutory requirements referred to above that are applicable to federal departments and agencies, but not to the IPCC.

Similarly, the draft guidelines need to better explain the roles of the CCSP, CCSP Principals, and working groups and various committees and subcommittees vis-à-vis the lead agency (or agencies) and the authors for each product.   The CCSP, the Principals, the groups, etc. seem to have a review/approval role as well as a drafting and coordination role, both of which seem to overlap with the responsibilities of the lead agency (or agencies) and the authors.   However, except for the selection of authors, the role of the lead agency (or agencies) is not well-defined, and they do not appear to have any role in the review/approval process, which is unusual.

It is our understanding that in the case of a number of products, the lead agency (or agencies) will not, unlike the IPCC, have the luxury of taking more than two years to produce a final product; that they are to collaborate with interagency groups and committees; and that their product, also unlike the IPCC, is not to be based solely on peer-reviewed literature, but will include, as noted in chapter 2 of the Strategic Plan, research results.

Indeed, chapter 2 explains that “[i]ntegration of research from agency programs and research elements is an essential component of the development of synthesis products and assessments that address CCSP goals.”   For each goal, the chapter includes “Examples of Key Research Activities” that “are not meant to constitute exhaustive lists” and explains that each of chapters 3-9 “describes research questions; provides an overview of the current state of knowledge”; and, among other things, “outlines milestones, products, and benefits from the research.”   The “completion dates” or “milestones” “range from 2004 to 2007 (greater than 4 years),” which are similar to the 2-4 year targets for the 21 listed products for chapter 2.   However, the guidelines apparently are not applicable to the research “products” in chapters 3-9, and they do not discuss how and to what extent the lead agency (or agencies) and the authors are to integrate such research results in the products.   If, as the Strategic Plan suggests, such “integration” is important and, as Dr. Solomon points out, “rushed reports pay the price in quality,” at the very least the prospectus should address how this “integration” is to be reasonably achieved.   It should include reasonable limits, as appropriate, on the research to be integrated, considering the 2-4 year targets established for the 21 products, although it is our understanding is that these time frames or milestones are not rigid deadlines.

We also note that the guidelines and the Strategic Plan apparently do not provide an opportunity for stakeholders, like EEI, to offer suggestions, possibly as part of the prospectus review, for additional issues to be considered by the lead agency (or agencies) in the development of the products.   Consideration should be given to providing such opportunity during the public comment period for the prospectus.

Second General Comment

The role of the lead agency (or agencies) under the draft guidelines needs to be clarified in light of applicable statutes.   In Phase IV, the guidelines state in one place that the lead agency (or agencies) “are responsible for developing products in conformance with these guidelines,” but adds that the lead agency (or agencies) “will” report “regularly to the CCSP Office” and in other Phases that CCSP Principles have a review and approval function.   In the case of the Interagency Working groups, Phase IV states on the one hand that they “may play a limited role related to scoping and/or reviewing,” but then indicates they may have a role in drafting the “reports.”   The guidelines need to define clearly these roles.   The lack of such a clear definition leads to conflicts or ambiguous authorities.

The Guidelines should reflect careful consideration of each point in the process of product development where there may be conflict with the roles and responsibilities of the lead federal agencies.   The involved federal agencies should have ultimate responsibility for the CCSP synthesis and assessment products.   For example, it does not seem appropriate for the guidelines to provide authority to the CCSP Principals (expressly or by reason of ambiguity)   that could undermine the authority of the federal agencies to carry out their respective responsibilities under applicable statutes.   It is fine for the CCSP Principals to perform functions that facilitate a comprehensive and consistent approach to product development, but it is quite another thing to cloak CCSP Principals with express or implied ability to second-guess federal agencies in carrying out their responsibilities.

Knowlton, Columbia University

-First general comment: A flow chart would be very helpful in clarifying the process steps for “Phase III: Review and Revision.”

-Second general comment: In 2 cases (CCSP Goal 2.1 and CCSP Goal 3.1) the US DOE is designated as the sole lead agency. These two topics (updating greenhouse gas emission scenarios and concentrations for 2.1; limitations of climate models including uncertainty, 3.1) are among the most controversial and most highly politicized aspects of climate change science. While DOE has expertise in energy utilization and emissions inventory analyses appropriate to Goal 2.1, there are other more biological components of emissions (forest cover, agricultural sources, etc.) for which other agencies would be appropriate as co-leads (EPA, USGS, NOAA). For Goal 3.1, EPA or NOAA aew suggested as a co-lead agency, owing to their expertise in risk assessment and uncertainty analysis. In both cases this additional expertise would contribute to DOE’s review strengths, and could help reduce possible public perception that DOE has been charged as lead agency on the two topics that are most sensitive with the current administration.

Neuman, Preserve Our Climate Coalition

"Scientists must feel confident that their participation in these assessment processes and the resulting products will be free from political interference."

"The CCSP's ambitious research plan overlaps considerably with the IPCC Fourth Assessment Report process. The US scientific community must be able to reasonably contribute to both efforts."

"Assessments should ideally be reasonably balanced on both results and process, such as engaging stakeholders through a variety of interactive approaches and ensuring that decision support needs are identified and met."

MacCracken, Climate Institute

1. The Proposed Guidelines Seem Unlikely to Meet FACA Requirements: Based on the discussions that went on in organizing the US National Assessment and its various activities, investigation needs to be done to determine whether the proposed guidelines meet the requirements of the Federal Advisory Committee Act (FACA). Any time that an agency or set of agencies plans to formally convene a designated set of individuals more than one time (as would seem to be necessary to generate the types of products that the CCSP is proposing), FACA requires that an advisory committee be formed and the FACA requirements on procedures and notifications be met. For the US National Assessment, the National Assessment Synthesis Team was formed as an advisory committee under NSF auspices on behalf of all of the USGCRP agencies, and the FACA guidelines for operating procedures applied (including about how members be chosen, etc.). It is not at all clear to me that the draft guidelines meet the FACA requirements for how the group would be legally required to operate.

Forming an advisory committee was not required for the regional teams as these were based in universities. In basing responsibility for an assessment in a university, however, the agency cannot then come in and exert the types of controls and reviews that are proposed in the draft guidelines—depending on the type of funding arrangement, up to full control must be given to the university team. For the sectoral teams, which were co-chaired by a scientist from a university and one from a federal agency laboratory or center, the FACA question did arise, but because the funding for these efforts was again generally handled through a university, an advisory committee arrangement was not deemed necessary. Again, control over the report was granted to the sectoral author teams and the types of reviews and controls envisioned in the guidelines would not have been allowed. For both the regional and sectoral reports, there was an expectation that there would be expert and stakeholder reviews, and this occurred, with the authors retaining control of the reports.

The issue of needing to form an advisory committee also came up with regard to the “blue ribbon” review committee that OSTP wanted to have watch over the process. Ultimately, this was done by having two members of the President’s Council of Advisers on Science and Technology (PCAST), which was chartered as an advisory committee, co-chair an ad hoc panel that also involved some other experts in the review process. The meetings of this ad hoc panel, which did not include a majority of the PCAST members, were permitted as long as the results of the meetings were presented to the full PCAST at an open, FACA-qualified meeting.

In that this issue of meeting the FACA guidelines was one of the matters raised in the first lawsuit about the National Assessment, I would very much urge that the legal counsels of the agencies be consulted to make sure that what is proposed meets the legal requirements.

The guidelines already start to point toward what might be a pathway for dealing with this FACA requirement. In particular, on page 1, lines 10-12, agencies are allowed to ask outside scientific entities to take on this task. This might be done under the auspices of one advisory committee that the CCSP agencies might jointly form (a suggestion of the NRC Graedel report for advising on the research effort)—again, it would have to meet all the various requirements. In that the advisory committee would now be the responsible entity for the reports, the agencies would need to cede many of their indicated responsibilities to this scientific advisory committee, such as being the final arbiter about whether the authors of the assessment reports had satisfactorily addressed matters raised in the various review processes, and whether an agency set of comments might be trying to force an unjustified change in the science. Such a path to dealing with the FACA issue would, of course, require a total rewriting of these guidelines and a conceptual change in how the documents are to be prepared.

While setting up all panels as FACA-qualified entities might seem an option, agencies are generally reluctant to set up advisory committees and the government-wide coordination office for advisory committees actively limits the overall number of committees and the number for each agency. Thus, doing the reports in this manner might well be problematic.

2. Over-involvement of the Agencies in the Process: While the intent of those who prepared the guidelines may have been more benign, the guidelines as written and read literally would seem to allow for significant over-reaching by the agencies, and in particular by political appointees. Involving the agencies so fully in the preparation of the report (e.g., going so far as to decide what references are and are not acceptable) and in the final reviewing and approval of the report will seriously compromise the scientific credibility of the report. Just as for the chapters and the technical summary in the IPCC process and for all the reports in the US National Assessment process, the reports of the scientific experts need to be accepted, with the agencies (or governments) certifying that they have gone through the proper process rather than attempting to get all agencies to agree to every word in the whole report.

For the US National Assessment process, we recognized that each agency would have its particular interests and priorities and perspectives, and that it would therefore be virtually impossible to get all agencies to formally approve the reports, especially in the event of a formal government-wide NSTC review. In addition to being impractical, imposing the type of a process proposed in the guidelines will legitimately bring the credibility of the reports into serious question (recall that many now in the Administration charged that National Assessment reports that did not even have a political review were politically controlled). The reviews by both the CCSP principals and even more by the NSTC mean that politically appointed officials would be controlling what is in what are said to be scientific reports. In my view, both personally and as president of an international scientific association, the guidelines would permit completely unacceptable interference with the independent expert role that scientists are expected to fulfill.

Were these reports to instead be the equivalent of the summaries for policymakers (SPMs) that are prepared by the IPCC, involvement of the levels of officials that are named would be acceptable, provided that it is made clear, as is the case in the IPCC process, that both the report’s scientific authors and the officials must be in complete accord with regard to the content. In particular, the IPCC is very careful to say that the scientific experts can refuse to accept any suggested change if it is not technically correct. Their SPMs are intended to be like the type of conversation a doctor (analogous to the scientist) and an interested patient (analogous to a government official) might have—the content of the discussion must be technically correct, but in terms that a non-expert can understand, covering the questions and issues that the non-expert is interested in, and framed in terms of levels of confidence, likelihood, and uncertainty with which the non-expert is experienced. So, jargon and concepts of scientific certainty and uncertainty need to be translated into terms that a non-expert is comfortable with, and the expert needs to be willing to convey their best judgment and prognosis—not simply hide behind obscure summarizations of the scientific literature. For this process to work, however, the process must be an open dialogue rather than being closed and done through an arms-length exchange of information controlled by just one side of the conversation.

Thus, it seems to me that in this context, the guidelines, as they are, are unacceptable, and they need to be refined to ensure that the scientific findings cannot be changed from those that are scientifically justified in the view of the expert authors, not the agency principals. This type of relationship should also be more comfortable for agencies, as, while individuals in agencies may be quite knowledgeable and have views, the research agencies should not be taking stands on scientific issues, but rather should be supporting the necessary, high-quality research needed to make sure that experts can come together to state key issues in a responsible and useful way. Agencies should be deciding if the evidence provided is sufficient to prompt action of various kinds, but should not be deciding that, for example, the climate sensitivity is really between 2.5 ºC and 4ºC rather than 1.5ºC and 4.5ºC, or something similarly technical. If the evidence is not sufficient, the agencies should be supporting the types of research to get the scientific community to be able to get to that point—and the scientific community can be a source of information about what types of research might be needed if provided some indication of the degree of confidence that is needed. But, to repeat, agencies should not be deciding the scientific questions; yet the way these guidelines are set up, that appears to be how the agencies are to be involved. The situation unfortunately seems somewhat analogous to the way the Catholic Church’s Roman Curia of several centuries ago was set up when dealing with Galileo; this approach for agencies coming to findings on scientific matters—that is, with the agency principals or higher levels in charge of deciding what can be said—has been widely recognized as a mistake in the past, and should continue to be considered unacceptable.

3. Overlap with the IPCC Process: Given the set of topics proposed, the indication that the panels appointed can be international, and the indication that the review will be such that the results will essentially be, or at least could be perceived to be, government policy (in that the results are being approved at the political level), the proposed activity and process is a serious intrusion into areas that have previously been delegated to the Intergovernmental panel on Climate Change (IPCC) by the governments of the US and the many other countries that are party to the UNFCCC. As such, the CCSP process would seem likely to seem offensive to other countries (will the US be coming to the IPCC with an open mind or with a foreordained position that it will not under any circumstances change?), will be seriously intrusive on the time of the scientific community presuming both approaches are to proceed (thereby meaning that the US scientific community will be able to do less than its best for both), and will cloud public and international understanding by returning to a situation where there is, at least in principle, more than one source of authoritative information (which is just the reason that the IPCC was formed to avoid). While it would be quite helpful to sponsor scientific reviews of various topics in addition to those that the IPCC is supporting, both beyond the area of climate change and in thinking about how the findings about climate change and its impacts apply to the specific situation facing the US and its environment and society, having the US Government producing what appear to be official (politically correct—or at least approved at that level) government positions on topics that the US and other countries have already delegated to the IPCC, will be detrimental to international relations, the scientific community, and public understanding.

4. Underinvolvement of Stakeholders in the Process: For many of the proposed topics, government leaders are not the only ones, and maybe not even the primary ones, interested in the outcome. For such stakeholders (e.g., state land and water managers, etc.), involvement in the process of preparing the assessment is absolutely vital. To really gain the confidence and participation of stakeholders, which is indicated as something to be encouraged in the research plan, the stakeholders have to have an opportunity for real involvement in the process, and this means with respect to every stage from outlining the need to reviewing and commenting on the final product. While there are currently a number of opportunities for such involvement and participation, that the final decisions are all elevated to the agency level, and with no indication that the reasons for any of the decisions will be publicly released and explained, the participation of stakeholders looks to be more pro forma than real.

With the Administration currently using the excuse that Executive Office comments on the CCSP’s draft plan are part of the government deliberative process and are therefore not available through the FOIA process, it is not at all clear that the comments made by the agency principals and agencies would be made part of the public record. For the public, and particularly for important stakeholders, to have confidence in the credibility of the assessments and syntheses, it is important that all inputs leading to changes in the text be open for all to see so that stakeholders (including the scientific community) can know that their findings are not being altered for political purposes. The research plan at present very much gives the impression that it will be the agency leaders who decide, for example, when some finding is certain enough to be useful to stakeholders rather than letting each stakeholder decide for themselves whether the level of confidence is enough for their particular situation. The present set of arrangements seems to be making the agency principals the “lords of the manor” with everyone else being court attendants or serfs. This is not the way to design a system that will win stakeholders to your side.

5. Reviews and Assessments by Other Entities: In proposing that agencies could collaborate with other entities on these studies, it is not at all clear what types of collaborations are being allowed. There is an indication that one example could be collaboration with the NRC, and this might be a very appropriate choice in some instances; however, in doing so, I would presume that the NRC would insist on its complete independence from agency involvement in the process of the choice of authors, report content, the review process, etc. Mention is also made of doing studies with international organizations, and it may be that what might be envisioned is the Department of State collaborating with the IPCC. If this is the case, it should be realized that the IPCC process would then govern. Under IPCC guidelines, US agency and political oversight on preparation of the IPCC chapters and technical summary would be limited to submitting review comments so as not to interfere with the independence of the IPCC demanded by all the world’s nations. The IPCC properly ensures that the scientific expert authors have the final word with regard to the content of their chapters.

If such delegations of responsibility to the NRC and IPCC are to be considered acceptable, and I would encourage this, then it does raise the question of why these guidelines should be vesting control of assessments about other issues in the hands of agencies and political appointees, as is proposed.

6. Assessment Must be an Author-Controlled Process: The present guidelines seem particularly inhibiting, indicating that all that is going to be done is to prepare a report that is under the strong control of the CCSP principals. Experience has very clearly taught that assessment needs to be viewed not just as a periodic report, but, at least importantly, as a learning process for the participants (and one that often goes beyond preparation of one report). To really advance scientific understanding, the author teams need to be allowed reasonable flexibility so that what is learned as they are assembling information can help to shape how the questions are posed and explained. The present guidelines seem to impose a rigid and time-consuming oversight process that will stifle productive scientific efforts to expand understanding. The agencies need to pose the general question they want to have addressed and then let the experts in the field (and coupled fields) work to formulate and reformulate the detailed structure and questions in order to find the best way of summarizing expert understanding and insights. If instead, the agencies predefine the entire scope and then allow only the changes that they approve, the process will be stiff and unproductive.

7. An Active or Passive Assessment: Assessment is generally considered to be more than simply a review of the literature—if all one wants is a summary of what is in the literature, go to a professional writer/compiler. An assessment requires making judgments and integrating knowledge. At times there may not be enough information to make a judgment because the right type of analysis has not been done, even though all the information is there to do it. In such cases, it is not uncommon that an existing, peer-reviewed capability (e.g., a model or data set) might be used to evaluate or explore a somewhat related question or issue. The guidelines need to make clear that the scientific authors are empowered to undertake such additional analyses and evaluations (some may be as limited as replotting information in different ways, others may require using peer-reviewed models in alternative simulations, others may require comparing the results of different models on the same test problem, requesting an author to run an alternative case or to further explain some point, to inquire into why different authors might get different answers to the same question, etc.)—that is, the authors really need to be allowed to have the power to assess, synthesize, and evaluate. These assessments are only likely to be of real value if such an active type of effort to assess is provided for; limiting the authors to passively combining and condensing only what is in the literature will not move scientific understanding forward any faster than does simple publication of articles in journals and being provided a bibliography. To undertake such efforts will require some financial support, and this is likely to mean more than simply the cost of traveling to meetings. The program needs to find a way to support the time of the authors to actually do some of this work.

8. Review Process: The drafters of the guidelines are to be commended for wanting there to be a full review process, including, especially for important reports, a Federal Register review process. Reports in the area of climate change need to communicate to such a wide community with such a variety of experiences and interests that having a full review process is necessary to make sure that findings and conclusions are clearly expressed, that all relevant research and evidence has been considered and evaluated, that experts and stakeholders both have opportunities to indicate their concerns, and that the right questions and issues are being posed. While such openness can make the process somewhat burdensome to authors, it has been my experience that the IPCC review process and the National Assessment review process, both of which involved multiple stages, led to significant improvements of the respective reports at all stages along the way. It has been my impression that doing anything less, and this includes doing only the very limited review that is carried out in confidence by the NRC, does not ensure that all of the various aspects that need attention get the attention that they deserve. For reasons of credibility, ensuring that everyone who might be interested has an opportunity to comment is also critical, and the attempt of the draft OMB guidelines on peer review to ensure adequate review make a mistake, in my opinion, in proposing to limit the set of potential reviewers—it should not matter from where a comment comes; what should matter is whether a comment makes a sensible point or not. So, I would encourage the CCSP leadership to revise the guidelines to ensure that reports are exposed to wide review, that all review comments are made public, and that the responses of the author teams are prepared and promptly made public.

9. Public Availability of All Information: If indeed the intent is to ensure the most credible summarization of an issue or question, then the full process must be open and documented. This will necessarily be accomplished if the FACA guidelines are applicable, but whatever system is utilized, there needs to be full public disclosure of the various drafts and of all comments made (including whom they are made by) and full documentation of why changes are or are not made. The IPCC has pioneered this effort, and this was done for the National Assessment process via the FACA and FOIA process requirements.

10. Time Required for an Assessment or Synthesis, and Narrowing the Set of Products to Subject to this Process: Given the structure that is being set up, the many stages and review requirements, this is going to be a quite time-consuming process. Even with diligent work, it seems unlikely that the various steps could be gone through in less than two years for any significant topic. For this reason, it really does seem that such efforts should be limited to the larger and more encompassing of the various topics, leaving the scientific reviews and syntheses on narrower issues to alternative mechanisms that do not necessitate such intense agency involvement. For example, while it would be nice to have a summary report on the detection and attribution of climate change, or on temperature change in the lower troposphere, such reviews should be left to the IPCC process or to articles prepared for journals. On the other hand, topics such as developing scenarios for the United States, much less for the world, do need a full effort that involves a wide array of inputs and reviews. And of course, the next assessment of the importance of climate change (or other aspects of global change) for the US should be done through such a structured (but open) process. Thus, I would urge the CCSP to select only a few reports from its list of near-term deliverables to be subject to the guidelines that are finally adopted. On purely scientific issues, leave out all the agency involvement and encourage the scientific community (including the IPCC) to instead carry out the synthesis, as is done today.

Making thoughtful choices of topics is also going to be important because it is very unlikely than any of the proposed assessments will completely and finally resolve an issue. As for the IPCC assessments, there will be a need to go back over a topic again and again as scientific understanding advances. Such assessments may not be needed every 2 or 4 years, but with new topics likely to arise and the need to update and redo various of the assessments, keeping the set of assessments to a manageable number will require very careful selection.

Moser, NCAR

1. While a clear delineation of rules governing the production of CCSP synthesis and assessment (S&A) products is welcome, it is yet again disappointing that the guidelines reflect only (but strongly) a traditional focus on products while understating the process component of assessment efforts. It is as if the CCSP had not learn from the First US National Assessment (though the shear fact that you define rules before engaging in any further synthesis and assessments (S&A) is progress!). It is as if you had not listened to the NRC Final Review of the CCSP Strategic Plan, which stated repeatedly that the lessons learned from the NA have been entirely neglected in the plan. I suggest several ways (here and in the specific comments section below) to improve on this overall impression:

1. Include an explicit statement that lead authors have the liberty to engage stakeholders in additional ways to the ones delineated in these draft guidelines (for examples see the NRC Review);
2. Greatly improve on your ideas about communication. Currently, this is something organized and implemented apart from the S&A process, conducted by lead agencies, and only at the very end of the process. That’s too late to begin, and raises serious doubts as to whether the product will be effective decision-support (see NRC Review, Chapter 2, “Decision Support”

2. The document current lacks any specific guidance to lead authors on linking the product with CCSP goals that each product is intended to have. The guidance on p.1 is far too vague and should be improved. This will give Congress (the funders!) and the general stakeholder public some confidence that the products will actually reflect and respond to the identified needs in the Strategic Plan.

3. The guidelines need to emphasize more strongly that and how S&A products will respond to the specific, and purposefully identified needs of decision- and policy-makers. This comment echoes the NRC recommendation in its review of the Final Strategic Plan (Chapter 2, “Synthesis and Assessment Products”). Again, this obviously can only be done through careful, sincere and intentional (and frequently only in face-to-face) interaction with the decision-makers the product is meant to serve, and should be done from the get-go, not only at the very end of the process. What is needed in these guidelines is greater specificity (through examples) and simply greater freedom for lead authors to conduct full scoping exercises. Many stakeholders will not read announcements on the CCSP website or in the Federal Register, thus mere public comment periods will not satisfy the need to engage, jointly explore and learn what stakeholders truly have at stake, and what their information and decision-support needs are. As the NRC Review stated:

“An additional benefit of conducting assessments is that they can serve to build and sustain constituencies, educate stakeholders, and build capacity in affected communities, while ensuring that communication channels between the scientific and decision-making communities remain effective avenues for decision support.”

To truly accomplish this, more than written input from stakeholders is required. Trust-building through direct involvement throughout a process is the key to success. Thus, these guidelines should reflect encouragement of such interactive processes.

4. Shielding from political influence of scientific S&A products breaks down at final stage of review process. Again, my comment echoes a recommendation made by the NRC in its review of the Final Strategic Plan (see Executive Summary Recommendation on “Maintaining Scientific Credibility of the Program”). The two main ways in which independence from political motivations breaks down are:

1. Government experts/scientists as peer-reviewers – by these rules – are not free and independent from political pressures (see Specific Comments below, incl. suggestion on how to fix this).
2. The final reviews by CCSP Principals and the NSTC (especially with its closeness to the White House and cabinet members) are completely unspecified and offer ample opportunity for political twists on scientifically peer-reviewed S&A products (see Specific Comments below and recommendations on how to fix this).

Many scientists – especially during a Bush Administration – will be reluctant to participate in any synthesis/assessment process unless you can guarantee freedom from interference of the political process. This is obviously highly problematic, given the ambitious nature and goals of the CCSP.

5. As the NRC warned, the ambitious goal of producing 21 S&A reports over the next four years may well lead to major capacity problems or at the very least assessment “fatigue” (see NRC Review of Final Strategic Plan, Chapter 2, “Synthesis and Assessment Products”). Heeding this warning and accounting for that very likely problem, the guidelines should include a statement that gives the lead agencies, in consultation with potential lead and contributing authors, the leverage to make a recommendation that only limited, scaled-down, partial or preliminary S&A products be produced, including a clear recommendation at what time a fuller S&A product can be expected.

Mutter, The Earth Institute at Columbia University

I was very pleased to see that the final draft of the US CCSP Strategic Plan included most of the comments and suggestions submitted during review of the first draft by members of the Earth Institute at Columbia University. I greatly appreciate you using our input. I was particularly pleased to see the increased emphasis on capacity building and partnerships with developing countries.

I would, however, like to encourage you to recognize and make appropriate emphasis in your efforts going forward that climate variability and change, regardless of its origins is sure to have disproportionate effects on poorest nations of the world. In these countries millions of people live on the edge of survival, barely meeting basic needs for life. Even small changes that could be managed in rich countries will send these poor regions into disaster and collapse. Humanitarian crises could follow. In the rich world, where our needs are met and considerably exceeded, we make the largest contribution to climate forcing through greenhouse gas emissions but have adaptive capacities that greatly exceed those in poorer countries.

The leadership position that we must take in climate science here in the United States must focus efforts on understanding the effects that climate change will have on poorer countries, in building adaptive capacity there and in mitigating the effects.

As the CCSP moves forward with the Synthesis and Assessment Reports listed in chapter 2, it is extremely important that the very great differences in the way climate variability does and will effect rich and poor countries is acknowledged and addressed in a focused program of directed research. This can be done by obtaining input from developing countries, themselves and from organizations that work in those countries.

These issues were recently tackled in depth at two major conferences hosted here at Columbia University: the State of the Planet and Earth’s Future: Taming the Climate, and were summarized in the State of the Planet Consensus Statement. You can find more detailed information about the results of both of these events, which brought in experts from around the world, at our website.

The Earth Institute at Columbia has as its mission the mobilizing of science to address the needs of all humankind but especially those of the poorest who have such limited capacity to deal with their own daunting problems.   We are ready to assist in any way as you go forward with the CCSP and CCRI initiatives.

Parkinson, NASA Goddard

The guidelines are readable and clear. My main recommendation would be to add a Phase V along the lines of:

"Phase V (as needed): Revision of Products Post Initial Release.

Recognizing that once a data product begins to be used, flaws are often identified and it becomes clear that an improved product is desirable, a CCSP-sponsored product will be able to be revised and submitted for a new scientifically-rigorous peer review. The revised product will have a new version number associated with it, to distinguish it from the original product. If the revised product is produced by the same individuals as the original product, it will only need to repeat Phases III and IV, but if it is produced by a different group, it will need to go through Phases I-IV."

Prather, University of California, Irvine

The draft guidelines for the CCSP Synthesis and Assessment Reports have an excellent design.   It follows in many ways the IPCC process and has a good combination of participation and external review. The only question I had was:   What does publish mean exactly? Although the backup science will appear in regular journals and some of these reports could readily be published in the standard peer-reviewed literature as review papers, I think that CCSP will need to establish a more reliable and uniform method of publication with a regular publisher.   As planned, the reports (like the IPCC) will be more thoroughly reviewed than any typical scientific paper and hence would not need to be forced into the peer-reviewed journals for credibility.

Solomon, NOAA Aeronomy Lab, CO

I have a series of general and important comments that cut across the document and will require major revisions to the document in many places.  

 1) The goals and audiences of the reports need to be clearly identified and presently are not.   This gives scientists a sense of purpose, and an understanding of who the product is addressed to. CCSP refers to decision makers and the public as audiences, but this is too generic.   What decision makers?   Which segments of the public?   How would the public be reached broadly (rather than narrowly)? Communication with the public would benefit from links to education -   e.g., high school and undergrad school teachers. CCSP would also benefit by making clear that qualified scientists themselves must be a key audience.    The reports will not be credible unless they are also credible to other scientists.   The revised document should clarify what is meant by decision makers and the public, should indicate that other scientists are also a key audience, and indicate what steps will be taken to reach each of these audiences.

2) There must be a clear demarcation of roles and procedures between the stakeholders/agencies/ government organizations and the scientists who produce the report, and the authors must have independence in their work if the reports are to be credible.   Agencies, CCSP principals, OSTP, or others should not have oversight, and they certainly should not have a right of final review.    Many people can and should participate in providing written review comments, but any oversight mechanisms should involve only distinguished scientists. Agencies should not have authority for appointing authors.    Reports will only be credible if a distinguished scientific committee is convened to choose the authors, perhaps with meeting support from an agency.     Such a committee should also be asked to make decisions regarding e.g. model results or unpublished work to be included, instead of the CCSP principals. Interactions and responsibilities of scientist/authors, reviewers, and audiences must be carefully designed (and documented) to maintain the independence of the science.    CCSP should commit to clarifying and documenting exact procedures and roles of all involved in CCSP assessments and procedures to be followed.   A formal 'procedures and roles' document should be produced before any work is begun.    All roles must be open, transparent, and public.

3) The reports should go through several staged and different rounds of review but the reviewers of each should not be the same.     All reviews should not be anonymous -- reviewer names should be given to the authors and made public.
Staged reviews.    There is a big difference between having just one review and having several of different types.   The IPCC system of developing a zero-order draft, informal review by well-known scientists, first order draft, broad expert review by a much wider group (including self-nominated individuals which thus includes the public),   and second order draft subject to a Government review means that the structure and content of the report is first determined by scientists.    The timing of the review involving governments (stakeholders) is key.    The CCSP review proposal does not appear to be staged to address its audiences sequentially.   I suggest changing the review plan to the above system, which allows for public comment, as well as input from stakeholders while at the same time allowing the scientists the time and process that is needed to reach mature drafts first using scientific inputs.

Non-anonymous reviews.    Anonymous reviews are appropriate for scientific journals, because they are not open to a wide spectrum of persons.   Rather, in the case   of a journal the editor of the journal (a senior and highly respected individual) carefully chooses the reviewers from well-qualified experts whose works are known.    This approach cannot credibly be used when any and all persons may comment on the documents.    The open review process that is planned makes it important that the reviewers be identified, so that all inputs are open and transparent.     It is not appropriate to use anonymous review here.

4) Scientific judgment should be the guiding force and final arbiter in the responses to the review process.     Senior distinguished review editors are a helpful role used in IPCC.   These individuals must be scientists of very strong reputation, and they should be asked to evaluate whether the authors have responded appropriately to the reviews.   Putting agency persons in that role would not be credible.   I suggest revamping the discussion of who will decide if the reviews have been properly treated to reflect the above.

5) The author teams must be given adequate time.    A rushed report pays the price in quality.   The document should make clear the commitment to take the time required to produce the reports to a high standard, regardless of the time required.

6) The assessment conclusions should be fully grounded in well-documented peer-reviewed literature, models, and datasets.   CCSP documentation would benefit from more clarity on how it will deal with this.   The procedures document referred to above should make clear what kinds of information will be included, and what would not be eligible for inclusion.

7) Stakeholder inputs should be provided in open yet formal forums.    Any feedback to the author teams from e.g., the agencies, the CCSP principals, OSTP, or others must be written down (not verbal) and made fully public.   No anonymous or informal feedback should be permitted from any of these sources, and this should be stated.

8) Clear science leadership must be provided by internationally known and respected scientists. CCSP should identify how the reports will be led -- I suggest a committee of designated “assessment science leaders” e.g., those leading the many different products.    These individuals should participate in a well defined, open, and formal assessment review feedback mechanism with agencies, OSTP and other stakeholders.

9) There must be explicit funding and technical support (TSUs) for the assessment process. The cost of TSUs is small compared to the hidden costs of volunteer scientist time, and the funding of the TSUs provides a critical contribution to the planning, compiling, editing, and management of the reports.    CCSP documentation should make clear that this kind of support will be provided.

Trenberth, NCAR

1. The proposal talks about Lead Authors.   However it is far from clear how the contributions from the lead authors are assembled into a chapter and how the chapters are assembled into a coherent document.   In IPCC the first is the responsibility of convening lead authors (CLAs) but no such people are designated here.   The second is achieved through plenaries and interactions through the CLAs.   Such mechanisms should be spelled out here.

 2. I am very concerned about these assessment activities in several ways.   Firstly they appear to be piecemeal.   But the main concern is that they will be occurring in parallel with IPCC.   The US is hosting IPCC WG I, and it requires major resources to carry out an assessment.  These are not only travel and organizational resources, but also huge commitments of scientists who are unpaid to do the assessment and writing.   The IPCC already is a huge burden on the community but has advantages of being an open international process.    The CCSP assessment will never achieve the credibility of the IPCC outside of the United States.   In my view it is essential that the CCSP assessment activities should be very clearly and explicitly related to the IPCC or they should not be done.   The timeline and how they relate to IPCC must be specified.   In fact the CCSP assessment, if timely, can be very useful input into IPCC.    If the activities are in fact in parallel then it has the potential to undermine IPCC and CCSP at the same time.   Also if they are separate the burden on the community is too great and the cost in terms of lost research at the expense of assessment will be huge.   It is very difficult to make substantial progress on the science while at the same time assessing the science.   I.e. this seems like too much of a good thing.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

It is important that the assessment products do not only have full scientific credibility, but also that they are perceived to have credibility throughout the international scientific and policy communities, otherwise they will not be used to underpin practical decisions on climate policy and further research. This being the case, it is important that the widest possible participation of independent experts is encouraged, both in the drafting and the review stages.

In the interest of making the process accessible, we suggest that a mailing list is established so that interested parties can be notified when report drafts are published on the CCSP websites.   

Winstanley, Illinois State Water Survey, IL

A critical component of the review process is the review of the reports by independent experts. The draft guidelines state that peer reviewers are to be selected on the basis of scientific and technical expertise, and that it is sufficient simply to include in the report reviews any previous involvement of reviewers with the lead agency(ies) or issues under consideration. In view of the critical nature of independent reviews, the above criteria and reporting requirements are necessary, but not sufficient to establish independence. I recommend that additional criteria be established for selecting independent reviewers. Such criteria might include not being related to the authors and/or program managers, not having worked together, not having co-authored reports, not being dependent upon involved agencies for support, and being otherwise free from influence of the outcome of the report.

Specific Comments

Page 1, Lines 7-14: The process described in this introduction and throughout the document is ad hoc, with a different mix of agencies and external involvement for each synthesis and assessment product.   We suggest that CCSP reconsider the National Research Council recommendation to establish a standing advisory body charged with independent oversight.   We believe such an advisory body would make the production of synthesis and assessment documents more efficient and consistent.   It would also ensure the input of external experts.

Abbott, NASULGC

Page 1, Lines 9-12

The Introduction provides that the lead agency (or agencies) “will collaborate” with CCSP working groups and the National Science and Technology Council subcommittees, without explaining how that collaboration would work and what impact it could have on timely completion of the products.   It adds that the collaboration must extend to “other national and international entities,” which is a rather far-reaching requirement, particularly since there are all sorts of such entities.   At the very least, this idea of collaboration with such entities needs more criteria and direction.   In its present form, it should be deleted.

Edison Electric Institute, Fang

Page 1, line 11: It is not at all clear why collaborations should be limited to national and international entities. Depending on the matter at hand, one would think that involvement might be with a much wider range of possible entities, including ones that are bilateral, regional, and even local. This is especially the case given that the CCSP research plan indicates the need for place-based studies; one could imagine undertaking assessments about particular regions, etc. It is also vital to getting buy-in of groups in various regions to base studies in that region, so the guidelines need to have a provision for coordinating with local and regional groups.

Michael MacCracken, Climate Institute

Page 1, Lines 13-14: The Introduction to the draft Guidelines quite properly directs that in order to “ensure scientific integrity and credibility, each deliverable will be produced in accordance” with the guidelines.   However, the sentence beginning on line 13 of the Introduction appears to undercut that directive in stating that “specific implementation” of the guidelines “will vary from product to product.”   It is a very broad and open-ended statement that is susceptible to wide and varied interpretation by the lead agency (or agencies), the authors and others.   It lacks any criteria for its application and any degree of oversight, and apparently could be applied in any phase of the guidelines without limitation and without any public knowledge.   It is also unclear to whom the provision is directed.   In addition, it appears inconsistent with the provisions of Phase I of the guidelines, which calls for a prospectus that is subject to public comment and then “finalized” and “posted on the CCSP website,” and which then provides (in the beginning of Phase II) that the lead authors “will prepare the product according to the process described in the prospectus.”  

There is sufficient flexibility in the guidelines – which are not rules or regulations – to enable lead agencies in applying them to each of the 21 products to recognize and accommodate, where necessary, various needs in their development.   We urge deletion of the sentence, or in the alternative a revision that would require identification of any needed variance and the reasons why it is needed as part of the draft prospectus required under Phase I of the guidelines, which is subject to public comment:

However, a lead agency (or agencies) may, in consultation with the IWGs, identify and explain in the prospectus any unusual circumstances where application of some aspect of the guidelines to a product may justify, consistent with such assurance of integrity and credibility, some form of limited modification.

Edison Electric Institute, Fang

Page 1, Lines 18-21: The first sentence of Phase I of the Guidelines provides that for each “product” the lead agency (or agencies) “should involve the scientific community and the public in a scoping process,” which “should culminate in preparation and review of a product prospectus.”   We have several concerns with this provision.  

First, the term “scientific community” seems to single out only one group of experts (i.e., government and non-government scientists) to the exclusion of technical experts – such as engineers, economists and sociologists – who, in the case of some products, would seem to have much to offer.   While it is good to mention the scientific community, it is also encompassed by the term “public.”   Further, it is equally appropriate to refer to, and specifically include, other experts and stakeholders in the scoping effort, as well as the general public.   As in the case of the IPCC, authors and contributors often are composed not only of scientists but also of a wide range of other technical experts.   For this and other public comment opportunities on the prospectus, the CCSP could establish and maintain a “list server” that would automatically send out in a timely fashion an E-mail to alert scientists, technical experts, stakeholders and the public of that opportunity.

Second, the sentence does not indicate how this involvement with scientists and the public in the “scoping process” is to be accomplished.   The terms “scientific community” and “the public” are quite broad and leave open how and to what extent each will be informed of the opportunity to be involved in the process.   There is no provision for Website or Federal Register notice.   Later in Phase I, provision is made for public comment on the draft prospectus.   However, that occurs after the scoping process.  

CCSP Director Richard H. Moss gave a presentation on the guidelines at the April 8, 2004, “Workshop on Issues in Global Change” held by the Global Change Coordinating Committee at the National Academies, which included a table on the “Status of Deliverables Scheduled for Completion Within 2 Years.”   The table indicates that in the case of some products, the prospectus (which is described in Phase I of the Draft Guidelines) has already been drafted or is “being drafted,” and in other products, the “scoping” that precedes the prospectus (also according to Phase I) has been completed or is “in progress.”   These activities apparently are taking place internally at the lead and supporting agencies and on an interagency basis with the CCSP.   Apparently because the timeline for these products began last October, Moss indicated that there would need to be “simultaneous review” of the guidelines and the several prospectuses referred to above.   However, he does not indicate in his comments to what extent, if any, involvement of the “scientific community and the public” occurred in the process or whether, after the guidelines are finalized, those efforts will be revisited in light of the guidelines.   It is unclear how initiation of the product process can begin in advance of finalization of the guidelines.   In the case of these products, the 2-4 year period could be viewed as beginning from the date of finalization of the guidelines, rather than from last October.    

Third, “should” on line 18 suggests that involvement of the scientists and the public might have a discretionary tone to it, leaving the impression that a lead agency could decide on a product-by-product basis about whether to involve either or both.   There is no good reason for such discretion.   Uniformity is the best approach.   This problem with “should” exists throughout the guidelines.   In some cases, “will” is used and in at least one instance “shall” appears.

Therefore, we urge that on lines 18 and 20 “should” be changed to “will” (or “shall”) in both sentences; on line 19, change “scientific community” to “scientific/technical community” and “public” to “stakeholders and the public”; and on line 18 after “involve”, insert

“, through the lead agency Website,”.

Incidentally, the term “stakeholders” is defined in Annex D of the Strategic Plan.   That definition is broad enough to include the entities that we listed above for that term.  However, if that is not your understanding, the guidelines should clearly include them.

Edison Electric Institute, Fang

Page 1, ln. 18-33: This section currently reads like an invitation to explore a tiny wiggle room. We have already defined the problem and the product (for you) and all you get to do is “further refine” it. This section should sound and be far more inviting for stakeholders to truly shape the product. This section should reflect the CCSP’s sincere openness and responsiveness to decision-support needs (whatever they may be). The bulleted list should include (i.e., be more explicitly demanding specifics on) what specific CCSP goal(s) and specific user/decision-support needs this product will respond to. The fourth bullet in particular should be reworded to be more inviting and permissive of a broad range of forms of open stakeholder involvement. “If needed” in parentheses suggests something exceptional, something tolerated but not really desired. Touché! This should be replaced with “as deemed appropriate and desirable”, and add examples of the kinds of ways in which stakeholders could be involved. You have a VERY traditional and narrow definition of stakeholder involvement! (See NRC Review of Final Strategic Plan for multiple additional examples).

Moser, NCAR

Page 1, line 30: Having each report possibly develop its own framework for expressing uncertainties seems very likely to greatly confuse the public. It would be much better to try to arrive at some more uniform approach for communicating levels of confidence and uncertainty in scientific understanding. In this regard, both the IPCC Assessment process and the National Assessment process have utilized a lexicon to express levels of confidence. Although similar, there are differences between these systems. One of the research recommendations in the National Assessment report was to encourage research on the strengths and weaknesses of these and other approaches (e.g., how effective were they or weren’t they in conveying information to the public, government leaders, the media, etc.) and to work to find a better system. IPCC is calling a meeting to consider further possibilities, but I am aware of no similar effort within the US to try to find ways to bridge the communications gap between the frameworks for uncertainty and risk used by the scientific community (and they vary across fields) and the more qualitative framework on which the public seems to rely.

Michael MacCracken, Climate Institute

Page 1, ln. 35: Nowhere do you define “CCSP Principals” – it is not clear who they are, how they are chosen, what their relationship is to the lead authors or reviewers, etc. All that needs to be stated clearly somewhere in this document.

Moser, NCAR

Page 1, Lines 35-40

The process laid out in this paragraph seems overly cumbersome and is likely to be time-consuming, particularly in the case of products with a target completion of two years after October 2003.  

First, it calls for two approvals of the prospectus by the “CCSP Principals.”    One is to take place before there is peer review and public comment and the other after that review and comment, although there is no procedure stated for submitting the draft prospectus to these Principals for either approval.  

Second, it includes a requirement for peer review.   However, that is premature because, according to Phase III of the draft guidelines, the prospectus is to describe the “review process” for each product.   Until the prospectus has gone through the public comment period and has been approved by the CCSP Principals, this process does not exist at Phase I of the guidelines.   Moreover, if there is to be a separate peer review, there needs to be a selection process for those reviewers as well.   That second review is not needed.

Third, the guidelines do not specify a time frame for the CCSP Principals to give or deny their approval.   In the case of the first approval, the guidelines call for them to “review” the “draft” in a “timely fashion,” which is not very meaningful.   In the case of the “final approval,” the guidelines do not even say that.   Again, given the tight time frames for the products, such an open-ended review process could be an obstacle to achieving the targets.

Fourth, neither the guidelines nor the Strategic Plan indicate who the CCSP Principals are, even though the term is used quite frequently in the guidelines.   It is our understanding that they apparently are program directors of the several lead and supporting agencies who may or may not be policy-makers or political appointees.   It seems odd that a federal agency would prepare the prospectus and then submit it for review to persons who are likely to be subordinate to those agency officials who oversee the preparation of the prospectus.  

It is important to reemphasize that this process, as described in Phase I of the guidelines, is begun by the “lead agency[ies].”    They are to conduct a scoping process leading to the development of a draft prospectus that is to address a list of specific points set forth in Phase I.   The draft prospectus is then subject to public comment.   Thus, we question the need for both approvals, given the fact that the prospectus is to be drafted by the lead agency and subject to public comment.   Accordingly, we urge that the paragraph on lines 35-40 be revised as follows:

Upon completion of the draft prospectus, it will be published on the CCSP website and in the Federal Register for a minimum of 30 days for public comment.   Upon consideration by the lead agency (or agencies) of the public comments, the draft prospectus with appropriate revisions will be submitted to the CCSP Principals for review, approval and publication on the CCSP Website.

As to the principals, at a minimum they need to be identified and some explanation given as to why they have been selected to grant approvals or denials over other line officials of the lead agency (or agencies).

Edison Electric Institute, Fang

Page 1, Line 38 - Responses to public comments should be included in the assessment.   Responses may be group by subject matter, however all subjects receiving public comment within the 30 day period should be addressed in the product assessment.

Neuman, Preserve Our Climate Coalition

Page 1, Line 42 through Page 2, Line 3

We urge these lines be revised to read as follows:

Selection of authors:   Such CCSP Website and Federal Register notice of the draft prospectus will also afford an opportunity for the public to nominate authors, contributors, and reviewers for the prospectus.   The lead agency (or agencies) are responsible for the selection of the authors, contributors and reviewers, who may be drawn from within or outside the federal government and the public and from outside the U.S. and independent of the public comment nominations.   Authors and contributors shall reflect a balance of scientific/technical expertise and points of view appropriate to the specific topic addressed in the product.

This revision makes clear that the Federal Register and Website notice for the comments on the draft prospectus will afford the public, in lieu of “interested parties,” an opportunity to nominate the authors, contributors and reviewers.   It also makes clear that the lead agency will select not only the authors but also the contributors and reviewers.   We assume that the lead agency will consider the nominees from the public, but will not be bound by that list.   In addition, the revision applies the same criteria for selection of the contributors as the authors.   At the same time, it deletes the requirement that they “be known for their scientific work related to the topic” expressed in the product.   That requirement could effectively exclude qualified authors and contributors, and it is uncertain how it would affect selection of authors and contributors for such products as those for CCSP Goal 5 in Table/Box 2-1 of the Strategic Plan.

We have retained the reference to “reviewers.”   However, it is unclear who they are or their roles.   Presumably, they are not peer reviewers, because the selection process of the latter is covered by Phase III of the guidelines.   This concept may be derived from the IPCC procedures, which provides for “Review Editors.”   However, the need for reviewers has not been demonstrated in this process.

Edison Electric Institute, Fang

Page 1, Line 42 – Page 2, Line 5

We applaud the public process for nomination of authors and contributors.   We urge that nominations be solicited from the university research community.

Abbott, NASULGC

Page 2, Line 4: Perhaps the choice of authors should reflect a balance of experts from within and outside the Federal government, in order for the products to have a broad ownership and demonstrate the participation of the scientific community as a whole.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 2, Lines 10-12

Phase II provides that the “drafting process will be coordinated” by lead authors with “national/international governmental and non-governmental entities.”   We presume that the lead authors are working for the lead agency (or agencies) and thus are subject to applicable federal law.   In this or any context, it is unclear what it means for these authors to coordinate with such entities and who decides which, if any, such “entities” are to have the favor of coordination.   The criteria for their selection are unexplained.   It is unclear whether this function to be part of the prospectus.   In addition, it is unclear what this does to the timetable for these products.   It is our understanding that the drafts are to be based on peer-reviewed literature and research, not such coordination.   This coordination should be left to public comment review and peer review.

Edison Electric Institute, Fang

Page 2, Line 14: Suggest that authors should primarily use published, peer-reviewed scientific literature in the drafting process.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 2, ln.14-20: It strikes me as rather odd, and likely politically motivated, to demand that lead authors – who supposedly are independent scientists – need to request permission from the (undefined) CCSP Principals to use “grey literature.” Don’t you trust those scientists (or the “independent peer review process” for that matter to be able to evaluate the appropriate use of this or any other literature? What really is the purpose of this stipulation? I recommend that the responsibility for evaluating the appropriateness of use of this type of literature be left to the lead author(s). You can specify that they pay particular attention to the appropriateness of its use, but not let any political appointees get in the middle of this decision. This then also eliminates the need for the ambiguously stated turn-around on this decision (“promptly” – p.2, ln. 20 – How long is that? This window can be totally misused if someone wants to!)

Moser, NCAR

Page 2, line 16: If experts submit contributions for consideration that are not published, it should be noted that they must be publicly available prior to the review of the draft of the assessment/report.

Michael MacCracken, Climate Institute

Page 2, Line 20: When responding to requests to use non-peer-reviewed material, the CCSP Principals should consider the balance of evidence used and ensure that it is primarily composed of peer-reviewed literature.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 2, Lines 22-25

First, the paragraph provides that the draft product “will include a scientific/technical analysis as well as a non-technical summary for the public.”   This appears to attempt to adopt the structure – generally described by Dr. Susan Solomon at an April 8, 2004, “Workshop on Issues in Global Change” held at the National Academies for IPCC reports – to the lead agency (or agencies) products.   It suggests that public, including stakeholder, input during the comment period should focus on the “non-technical summary.”   It also suggests that the public is not capable of understanding the underlying scientific/technical analysis.  

While we do not object to the product including such a summary, in light of the statutory provisions cited in our General Comments – particularly the Information Quality Act – attempting to draw what may be described as a bright line for the public between the underlying analysis and summary is inappropriate.   We also note that in addition to stakeholders the public includes scientists/experts who may not be selected as authors or contributors.   Under Phase III, provision is made for public comment on the entire product, not just the summary, which is quite appropriate.   Therefore, we urge that a period be inserted after “summary” on line 22 and that “for the public” on line 23 be deleted.

Second, the sentence beginning on line 23 provides that the “products should identify disparate views that have significant scientific or technical support.”   If such “views” have scientific support, technical support or both, there should be no question that they need to be included in the “products.”   However, “should” suggests that the lead agency (or agencies) or authors would have discretion whether to include those views.   There is no reasonable rationale for such discretion.   We urge that on line 23 “will” (or “shall”) be substituted for “should.”  

Third, the sentence beginning on line 24 indicates that the products should provide “confidence levels for findings, if this is appropriate to the product.”   We assume that this refers to the use of probabilities relative to uncertainties.     However, the term “confidence levels” is not explained, nor is there an indication as to who determines that such levels would be “appropriate to the product.”   It suggests reliance on subjective judgments and lacks any provision for transparency.   This is a very important issue that deserves more consideration and greater direction and criteria than just this single, rather open-ended sentence.   For example, a basic question exists as to whether any probability or confidence level assertion should be made relative to these products.   The Strategic Plan calls for 21 products, each with its own lead author (or authors), and this could result in 21 different approaches and decisions.  

We are concerned about this provision.   Every   “prediction” or “projection” of future climate phenomena potentially attributable to human activities depends, in the first instance, on assumptions as to future levels of anthropogenic emissions of greenhouse gases and aerosols.   The myriad of driving forces that determine these emissions (including economic, demographic and technological factors, and non-climate government policies) and their interrelationships necessarily defy rational probability assertions, and the inherently subjective nature of any guesses as to how future emissions paths will unfold is accentuated by the increasingly speculative nature of assumptions about the underlying, driving forces over time horizons as long as a century.

Lack of scientific justification for asserting the probability of occurrence of any particular level of future emissions over long time periods is sufficient reason, standing alone, for rejection of efforts to have the CCSP products declare “confidence levels” for the occurrence of future climate phenomena.   Of course, there are additional concerns about the substantial uncertainties regarding factors such as climate sensitivity and other issues with respect to global and regional climate modeling exercises that render unappealing assertions about such confidence levels.   Terms such as “likely” or “unlikely” are either meaningless or capable of creating unjustified impressions unless they are accompanied by a standard, agreed set of quantitative definitions, but that is precisely what is inappropriate.

It is our understanding that the IPCC’s Third Assessment Report includes such “levels” for Working Groups I and II, but not Working Group III.   A September 5, 2003, paper titled “A Concept Paper for the AR4 Cross Cutting Theme:   Uncertainties and Risks” by Martin Manning and Michael Pettit discusses the “treatment of uncertainty in the TAR.”   It noted that Working Group I “adopted a different seven-level scale to characterize confidence” than the “five-level confidence scale” used in Working Group II and stated that “[i]n retrospect it appears that the use of specific language (words such as likely or low confidence) to describe probability ranges can be misleading or confusing and this aspect of describing uncertainty needs to be reviewed.”   We urge deletion of the sentence, as well as the reference to “conference levels” in Phase I regarding the prospectus.

Edison Electric Institute, Fang

Page 2, Line 23: Identification of disparate views that have significant scientific support should be consistent. This could be facilitated by provision of a definition of ‘significant’ support – for example, evidence of significant support could be demonstrated by one or more peer-reviewed supporting papers in the literature.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 2, Line 24: Reports on confidence levels for key findings should be consistent. All key findings should be identified, and confidence levels provided for all of them.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 2, line 24:    I suggest the phrase "provide confidence levels for key findings" be followed by "based on clearly defined methodologies."   Sometimes confidence limits can be only crudely guessed at and sometimes they can be rigorously defined; it is helpful to know which is which.

Barlow; AER, Inc.

Page 2, Lines 27-30 and Footnotes 1 and 2

This paragraph and the footnotes propose to implement the requirements of the Information Quality Act (which is section 515 of Pub. Law No. 106-554) by stating that the lead agency (or agencies) “should” provide to the authors the information quality guidelines issued by the Commerce Department (DOC) and NOAA and that the authors “should” develop the products in “accordance with these guidelines.”   Phase IV of the draft guidelines provides that each final synthesis and assessment product is to be “identified as a CCSP-sponsored product,” is to be “published in a consistent format to ensure that all deliverables are seen as part of the family of CCSP-sponsored” products, and “will indicate the Federal agency or agencies that led its preparation.”   In addition, the “products will include a statement that indicates the product was prepared according to the DOC/NOAA information quality guidelines.”   That proposal is inconsistent with the Information Quality Act and the Office of Management and Budget (OMB) final Guidelines, 67 Fed. Reg. 8452 (Feb. 22, 2002).  

The OMB guidelines apply “government-wide.”   They state that the Information Quality Act “directs” “agencies subject to the Paperwork Reduction Act” to issue “their own information quality guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information, disseminated by the agency” within one year after the OMB guidelines are issued.   It is our understanding that all of the lead agencies listed in the Strategic Plan have complied with this requirement.   However, the CCSP is not a federal agency.   Rather, as explained in Chapter 16 of the Strategic Plan, it is a “federal program” established in 2002 by the President as part of a “new Cabinet level management structure,” with “ultimate budget accountability” residing “with the participating agencies and departments.”   It reports to the Cabinet-level committee on Climate Change Science and Technology Integration.   Thus, presumably the CCSP has not prepared these guidelines, but rather relies on the DOC/NOAA guidelines, although apparently it is not part of DOC/NOAA.

Notwithstanding the provisions of Phase IV of the draft guidelines, the information quality guidelines of each of the lead agencies would, by law, apply to the products prepared by those agencies.   DOC/NOAA are the sole lead agency in a number of products, and their guidelines would apply to their products.   In other products, they share the lead agency role, and we presume that the DOC/NOAA guidelines could apply.   However, in the case of the remaining products, the DOC/NOAA guidelines could not be construed to apply.   Moreover, the Information Quality Act and the OMB guidelines unquestionably require that each of these agencies make their guidelines available to the authors of the products being produced by each such agency and require the authors to abide by them.   The statute does not allow the lead agencies to, in essence, ignore their own information quality guidelines.   Therefore, we urge that this paragraph and the footnotes be revised consistent with these comments.

Edison Electric Institute, Fang

Page 2, Line 32 – Page 4, Line 2:

The synthesis and assessment products produced by the CCSP will be critical inputs to the development of U.S. climate policy.   As such, it is critical that they have the highest level of credibility.   To ensure credibility, it is necessary that CCSP provide a transparent process for soliciting and responding to public comments on draft products.

The Draft Guidelines for Producing CCSP Synthesis and Assessment Products promise the opportunity for public comments.   However, they do not provide transparency on the response to these public comments.   The Draft Guidelines state:   

Lead Authors will revise the draft product to incorporate the public comments, as they deem appropriate, and the CCSP Principles will review the product a final time. (Pg. 3, lines 40-41.)

No independent check on the decisions of the Lead Authors is provided, nor is there any indication of mechanism for providing the CCSP Principles with the scope and disposition of public comments.   Reviewers providing comments during the public comment period can only infer how their comments were addressed when the final report issues.

We recommend that the CCSP develop a transparent procedure that will ensure that all public comments are given appropriate consideration, and that their scope and disposition is reported to the CCSP Principles before they complete their final review.   The procedure could be modeled after the one currently in use by the IPCC.   This procedure is detailed in a document titled: Procedures for the Preparation, Review, Acceptance, Adoption, Approval and Publication of IPCC Reports [PDF].  

The IPCC procedure involves the use of review editors whose functions include ensuring that all substantive review comments are afforded appropriate consideration.   Review editors are experts in the field covered by the report, but they are neither authors nor reviewers of that report.

IPCC applies it procedure by requiring a written record of the disposition of all review comments.   Lead Authors can respond to comments by making the appropriate change in their report or by providing a reason why no change is required.   The review editor then provides assurance in writing to the IPCC body approving or accepting the report that the Lead Author team has considered all comments and that all have been responded to in an appropriate fashion.

IPCC procedures also state:

All written expert and government review comments will be made available to reviewers on request during the review process and will be retained in an open archive in a location determined by the IPCC Secretariat on completion of the Report for a period of at least five years.

We recommend that CCSP adopt a similar procedure.

In practice, these procedures need not be cumbersome.   The documentation on public comments distributed to Lead Authors can also provide space for indicating how the Lead Author team responded to each comment.   The review editor should attend the discussion of public comments, which will allow him/her to advise the Lead Author team as to whether all comments have been handled appropriately.   Once the Lead Author team has completed its response to public comments, the review editor will be in a position to assure the CCSP Principals that public comments have been appropriately addressed.   Finally, the CCSP Library can act as an archive for public comments for the appropriate storage period.

Instituting this type of procedure will ensure transparency in CCSP review processes and credibility for CCSP Assessment and Synthesis products.   

O’Keefe & Bernstein, George Marshall Institute

Page 2, Line 34 – Page 3, Line 13

This paragraph of Phase III of the draft guidelines provides that the products “will have an appropriate and scientifically rigorous peer review,” that the “process” is the “responsibility of the lead agency,” and that it is to be described in the prospectus and “approved by the CCSP Principals.”   On April 28, 2004, OMB issued its “Revised Information Quality Bulletin for Peer Review” of “Influential Scientific Information” and “Highly Influential Scientific Assessments” for additional public comment.   69 Fed. Reg. 23230 (2004).   When the OMB revision takes effect later this year, it will apply to all federal agencies subject to the Paperwork Reduction Act, which includes all of the lead agencies.   These guidelines must conform to that revision when it does take effect.

Edison Electric Institute, Fang

Page 2, Line 35: Should provide a guide for the process used to choose independent experts, and for appropriate numbers of experts.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 2, Lines 35-36: In connection with the “review process,” Phase III calls for a rigorous per review and states that the process “will include scientific/technical review by independent experts.”   It gives the impression that there are two reviews, one by peer reviewers and one by experts.  It is unclear whether that is intended.   In addition, it is unclear what the word “independent” means regarding such experts.   Questions arise about who are they independent of and what they are independent of.   It is unclear whether this phrase is intended to exclude experts from agriculture, business, labor, industry or the environment.

Edison Electric Institute, Fang

Page 2, footnote 2 : Provide the url for the OMB guidelines as well.

Moser, NCAR

Page 3, Line 2: The review process should ensure a balance between independent and government associated reviewers.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 3, ln.6: You state on p.2, ln. 35-36 that the peer-review is to be conducted by independent scientists/experts. Here you include government scientists/experts as potential reviewers. In this day and age, government scientists can only be independent if they don’t care about their job security. As there are many highly qualified government scientists/experts, I suggest you include a statement here relieving them of conformity with government policy for the purposes of this INDEPENDENT peer review.

Moser, NCAR

Page 3, Line 6

The words “as necessary” gives the wrong connotation and suggests a limit on the experts selected.  What is needed for peer review is not just a “broad range” of expertise, but also a broad range of views.   Therefore, we recommend deletion of the qualifying words “as necessary” and insertion instead of “and points of view.”

Edison Electric Institute, Fang

Page 3, Line 20 through Page 4, Line 2

These provisions of Phase III of the guidelines provide that:   the “individual” peer reviews are to be “made available to the CCSP Principals when completed”; “lead authors” should revise the draft products to “incorporate the peer review comments, as they deem appropriate”; to the extent such comments are not incorporated, the authors must “prepare a brief explanation” for the Principals; the lead agency or (agencies) are to provide the “revised” draft product to the Principals; the Principals are to review the draft and approve it for release for public comment; thereafter lead authors are once again to review the draft and “incorporate” the results of the public comments; and the Principals are to review it a “final time” and give their final approval.

First, this portion of the guidelines raises the question of the role of the lead agencies in the preparation and approval of the products.   For example, the provisions for the lead authors providing to the Principals “individual” peer reviews and the “brief explanation” about not incorporating some “peer review comments” and for the lead agency (or agencies) sending to the Principals “draft products” for review and approval before public comment appears to make them nothing more than funding and resource sources, overseers of the process, and conduits to the Principals, who apparently are lead agency personnel.   Yet Phase IV provides that each product will indicate the agency that “led its preparation.”   Phase IV also provides that the lead agencies “are responsible” for developing products and for all phases of their preparation.

The wisdom of this process in unclear.   Since the lead authors are selected by the lead agency (or agencies), it would seem appropriate that it is the agency’s responsibility to receive the peer review materials and to review the draft product prior to public comment.   In addition, the sentence beginning on line 36 provides that public comment notices must clearly state that the draft product “does not represent agreed findings” of the “participating agencies” or that of the Principals, even though the Phase III provisions do not seem to provide any opportunity for “agreed findings” by both the agencies and the Principals.   Only the Principals are mentioned.

Second, the need to involve the Principals in the review of the product before it is noticed for public comment is not justified.   This review should be undertaken by the agency.   The Principals should only be involved after the public comments have been received, the lead authors have considered them, and the agency has reviewed the draft with the appropriate revisions.

Third, in the case of the peer review comments, the lead authors apparently have broad discretion whether to revise the draft product through application of the word “should” and the words “as they deem appropriate.”   Nevertheless, they must explain to the Principals any “aspects” of the peer review comments that they do not incorporate into the product.   However, the guidelines do not indicate what, if anything, the Principals should do if they disagree with the explanation.

Fourth, the sentence beginning on line 28 calls upon the Principals to, among other things, review the product to see if the summary “describes the findings in a context understandable to the public.”    That suggests that the public is incapable of understanding a summary that “accurately conveys the findings of the scientific/technical material and describes the findings.”   The reference to the public is unnecessary.

We urge the following revisions:   On lines 21 and 25 before “CCSP Principals”, insert “lead agency[ies] and the”.   On line 27, insert “for information purposes” before the period.   On line 28, delete “CCSP Principals” and insert “lead agency[ies]”.   On line 30, delete all after the second “findings” until the period on line 31.   On line 37 after “represent”, insert “an approved and final product” and delete the remainder of the sentence until the period on line 38.

Edison Electric Institute, Fang

Page 3, line 23: Suggest this should read “…revise the draft product in the light of the peer review comments, as they deem appropriate.” The comments might be of the nature of guidance or instructions, rather than wording intended to be incorporated.

UK Department for Environment, Food and Rural Affairs (Johnson-DEFRA)

Page 3, line 25: Suggest replace “incorporated” with “accepted” (to be consistent with comment above on page 3 line 23.)

UK Department for Environment, Food and Rural Affairs (Johnson-DEFRA)

Page 3, Line 25: Propose that the reasons for choosing not to incorporate review comments should be explained to reviewers as well as the CCSP Principals.

UK Department for Environment, Food and Rural Affairs (Oliver-DEFRA)

Page 3, Line 36: Change “30 days” to “60 days.”   Rationale: Given the importance of the products, the valuable input that may be obtained through public review, and the potential volume of the material to be reviewed, it would be preferable to extend the public comment period to a minimum of 60 days.   [See General Comments for further discussion.]

American Petroleum Institute, Feldman

Page 3, Line 40: Suggest this should read “… revise the draft product in the light of public comments, as they deem appropriate.” The comments might be of the nature of guidance or instructions, rather than wording intended to be incorporated.

UK Department for Environment, Food and Rural Affairs (Johnson-DEFRA)

Page 3, Line 40: Perhaps the lead authors’ reactions to the public comments – whether they are accepted, or rejected, with reasons – should be made available to those members of the public who commented?

UK Department for Environment, Food and Rural Affairs (Johnson-DEFRA)

Page 3, ln.41-43: Here is the place where the political independence of the review process and of the production of scientific synthesis and assessment products breaks down. Very dangerous section! At the VERY LEAST, you must make explicit here what the purpose and POWERS of the final CCSP Principals review is, and that of the NSTC review process is. Also specify what guards will be in place to protect against political influence at this final stage.

Moser, NCAR

Page 4, ln.10-12: It concerns me greatly that the CCSP office and lead agencies will develop a communications/outreach plan (a) independent of the lead and contributing authors, and (b) that they will do so only this late in the game. Again, this reflects the very traditional, very out-dated, and product-focused approach described and criticized in the general comment section. I recommend that this communications plan be developed at the start of the project, and be subject of the stakeholder scoping, expert peer review and public review processes.

Moser, NCAR

Page 4, Lines 26-38

This portion of Phase IV states that the “IWGs will be able to contribute significantly to the preparation of a deliverable” and lists, as one of their “potential roles,” drafting “sections of reports” (i.e., products), subject to “agreement” with the lead “agency[ies]”.  

The guidelines do not identify the IWGs or indicate their membership.   However, Phase II states, “The lead authors will prepare the product according to the process described in the prospectus.”

In addition, Phase IV specifies that the lead agencies “are responsible for developing products” and they are to “take responsibility for all phases of product preparation.”   In light of these provisions, it is unclear what is intended by these provisions concerning the IWGs.   Furthermore, it is unclear what benefit is achieved through drafting by groups.   The role of the IWGs needs to be reconsidered in light of the requirements for an approved prospectus.   If they are needed, their role should be set forth in the prospectus.