Appendix B: Consultation with Experts and Document Review
Monterey Bay National Marine Sanctuary

The process for preparing condition reports (and similarly, this update) involves a combination of accepted techniques for collecting and interpreting information gathered from subject matter experts. The approach varies somewhat from sanctuary to sanctuary, in order to accommodate different styles for work with partners. The Monterey Bay National Marine Sanctuary approach was closely related to the Delphi Method, a technique designed to organize group communication among a panel of geographically dispersed experts by using questionnaires, ultimately facilitating the formation of a group judgment. This method can be applied when it is necessary for decisionmakers to combine the testimony of a group of experts, whether in the form of facts or informed opinion, or both, into a single useful statement.

The Delphi Method relies on repeated interactions with experts who respond to questions with a limited number of choices to arrive at the best supported answers. Feedback to the experts allows them to refine their views, gradually moving the group toward the most agreeable judgment. For condition reports, the Office of National Marine Sanctuaries uses standardized questions related to the status and trends of sanctuary resources, with accompanying descriptions and five possible choices that describe resource condition (Appendix A).

In order to address the standardized questions, sanctuary staff selected and consulted outside experts familiar with water quality, living resources, habitat and maritime archaeological resources in the estuarine, nearshore, offshore and seamount environments. A few different approaches (e.g., small group meetings, conference calls, email and individual meetings) were used to get expert input on the questions, depending on the availability of experts (a list of experts who provided input is available in the Acknowledgement section of this report).

In these meetings and calls, experts were introduced to the questions and then asked to provide recommendations and supporting arguments. In small group settings and conference calls, the group converged in their opinion of the rating that most accurately described the current resource condition. In individual meetings and email correspondence, the sanctuary staff considered all input and decided on status and trend ratings. In all cases, draft status and trend ratings along with supporting narratives were made available to experts for individual comment.

Experts were also consulted to assign a level of confidence in status and trend ratings by: (1.) characterizing the sources of information they used to make judgments and (2.) their agreement that the available evidence supports the selected status and trend ratings. The evidence and agreement ratings were then combined to determine the overall confidence ratings, as described in the table here.

Step 1: Rate Evidence
Consider three categories of evidence typically used to make status or trend ratings: (1.) data,(2.) published information and(3.) personal experience.

Evidence Scores
Limited	Medium	Robust
Limited data or published information, and little or no substantive personal experience.	Data available, some peer reviewed published information, or direct personal experience.	Considerable data, extensive record of publication, or extensive personal experience.

Step 2: Rate Agreement
Rate agreement among those participating in determining the status and trend rating, or if possible, within the broader scientific community. Levels of agreement can be characterized as "low," "medium" or "high."

Step 3: Rate Confidence
Using the matrix below, combine ratings for both evidence and agreement to identify a level of confidence. Levels of confidence can be characterized as "very low," "low," "medium," "high" or "very high."
chart of rate confidence

An initial draft of the update, which was written by sanctuary staff, summarized the new information, expert opinions and level of confidence expressed by the experts (who based their input on knowledge and perceptions of local conditions). Comments, data and citations received from the experts were included, as appropriate, in text supporting the ratings. This initial draft of the update was made available to contributing experts and data providers which allowed them to review the content and determine if the report accurately reflected their input, identify information gaps, provide comments or suggest revisions to the ratings and text. Upon receiving those comments, the writing team revised the text and ratings as they deemed appropriate. In some cases, additional review of certain sections, by those with specific expertise, was requested after revision. Sometimes, additional input on confidence scores was requested if the status and trend changed after those ratings had first been established in a small group setting.

In July 2015, a draft final report was sent to regional scientists for final review (listed in the Acknowledgemen’ section of this report). In December 2004, External Peer Review became a requirement when the White House Office of Management and Budget (OMB) issued a Final Information Quality Bulletin for Peer Review (OMB Bulletin) established peer review standards that would enhance the quality and credibility of the federal government’s scientific information. Along with other information, these standards apply to Influential Scientific Information, which is information that can reasonably be determined to have a "clear and substantial impact on important public policies or private sector decisions." The condition reports are considered Influential Scientific Information. For this reason, these reports are subject to the review requirements of both the Information Quality Act and the OMB Bulletin guidelines. Therefore, following the completion of every condition report, they are reviewed by a minimum of three individuals who are considered to be experts in their field, were not involved in the development of the report and are not ONMS employees. Comments from these peer reviews were incorporated into the final text of this report. Furthermore, OMB Bulletin guidelines require that reviewer comments, names and affiliations be posted on the agency website, http://www.cio.noaa.gov/. Reviewer comments, however, are not attributed to specific individuals. Comments by the External Peer Reviewers are posted at the same time as the formatted final document.

The reviewers were asked to review the technical merits of resource ratings and accompanying text, as well as to point out any omissions or factual errors. Following the External Peer Review, the comments and recommendations of the reviewers were considered by sanctuary staff and incorporated, as appropriate, into a final draft document. The final interpretation, ratings and text in the draft condition report were the responsibility of sanctuary staff, with final approval by the sanctuary superintendent. To emphasize this important point, authorship of the report is attributed to the sanctuary alone. Subject experts were not authors, though their efforts and affiliations are acknowledged in the report.

Estuarine Environment - Confidence Scoring Table

Question	2015 Rating	Evidence *(Limited, Medium or Robust)*	Agreement *(Low, Medium or High)*	Confidence *(Very Low, Low, Medium, High or Very High)*
Water Quality
Question 1: Multiple Stressors	Status: Fair/Poor	Not updated	Not updated	Not updated
Question 1: Multiple Stressors	Trend: Declining	Not updated	Not updated	Not updated
Question 2: Eutrophic Condition	Status: Fair/Poor	Robust	High	Very High
Question 2: Eutrophic Condition	Trend: Declining	Robust	High	Very High
Question 3: Risks to Human Health	Status: Fair/Poor	Not updated	Not updated	Not updated
Question 3: Risks to Human Health	Trend: Undetermined	Not updated	Not updated	Not updated
Question 4: Human activities and Water Quality	Status: Fair	Medium	High	High
Question 4: Human activities and Water Quality	Trend: Improving	Medium	High	High
Habitat
Question 5: Major Habitat	Status: Fair/Poor	Robust	High	Very High
Question 5: Major Habitat	Trend: Not changing	Medium	Low	Low
Question 6: Biologically-Structured	Status: Poor	Robust	High	Very High
Question 6: Biologically-Structured	Trend: Improving	Robust	Medium	High
Question 7: Contaminants	Status: Fair/Poor	Low	Low	Very Low
Question 7: Contaminants	Trend: Declining	Low	Low	Very Low
Question 8: Human Activities and Habitat	Status: Poor	Medium	Medium	Medium
Question 8: Human Activities and Habitat	Trend: Improving	Medium	Low	Low
Living Resources
Question 9: Biodiversity	Status: Fair	Medium	Medium	Medium
Question 9: Biodiversity	Trend: Not changing	Medium	Low	Low
Question 11: Non-Indigenous Species	Status: Poor	Medium	Medium	Medium
Question 11: Non-Indigenous Species	Trend: Not changing	Medium	Medium	Medium
Question 12: Status Key Species	Status: Fair/Poor	Robust	High	Very High
Question 12: Status Key Species	Trend: Improving	Robust	High	Very High
Question 13: Condition Key Species	Status: Good/Fair	Limited	Medium	Low
Question 13: Condition Key Species	Trend: Undetermined	Limited	Medium	Low
Question 14: Human Activities and Living Resources	Status: Fair/Poor	Medium	Medium	Medium
Question 14: Human Activities and Living Resources	Trend: Undetermined	Limited	Medium	Low
Maritime Archaeological Resources
Question 15: Integrity	Status: Undetermined	Not updated	Not updated	Not updated
Question 15: Integrity	Trend: Undetermined	Not updated	Not updated	Not updated
Question 16: Threat to Environment	Status: Good	Not updated	Not updated	Not updated
Question 16: Threat to Environment	Trend: Not changing	Not updated	Not updated	Not updated
Question 17: Human Activities	Status: Good	Not updated	Not updated	Not updated
Question 17: Human Activities	Trend: Not changing	Not updated	Not updated	Not updated

Nearshore Environment - Confidence Scoring Table

Question	2015 Rating	Evidence *(Limited, Medium or Robust)*	Agreement *(Low, Medium or High)*	Confidence *(Very Low, Low, Medium, High or Very High)*
Water Quality
Question 1: Multiple Stressors	Status: Fair	Not updated	Not updated	Not updated
Question 1: Multiple Stressors	Trend: Declining	Not updated	Not updated	Not updated
Question 2: Eutrophic Condition	Status: Fair	Robust	Medium	High
Question 2: Eutrophic Condition	Trend: Declining	Robust	Medium	High
Question 3: Risks to Human Health	Status: Fair	Robust	High	Very High
Question 3: Risks to Human Health	Trend: Undetermined	Robust	High	Very High
Question 4: Human Activities and Water Quality	Status: Fair	Limited	High	Medium
Question 4: Human Activities and Water Quality	Trend: Improving	Limited	High	Medium
Habitat
Question 5: Major Habitat	Status: Fair	Robust	High	Very High
Question 5: Major Habitat	Trend: Declining	Robust	High	Very High
Question 6: Biologically-Structured	Status: Good	Robust	High	Very High
Question 6: Biologically-Structured	Trend: Not changing	Robust	High	Very High
Question 7: Contaminants	Status: Fair/Poor	Medium	High	High
Question 7: Contaminants	Trend: Declining	Medium	High	High
Question 8: Human Activities and Habitat	Status: Fair	Robust	Low	Medium
Question 8: Human Activities and Habitat	Trend: Undetermined	Robust	Low	Medium
Living Resources
Question 9: Biodiversity	Status: Fair	Robust	High	Very High
Question 9: Biodiversity	Trend: Not changing	Robust	High	Very High
Question 11: Non-Indigenous Species	Status: Good	Robust	High	Very High
Question 11: Non-Indigenous Species	Trend: Declining	Robust	High	Very High
Question 12: Status Key Species	Status: Fair	Robust	High	Very High
Question 12: Status Key Species	Trend: Declining	Robust	High	Very High
Question 13: Condition Key Species	Status: Fair	Robust	High	Very High
Question 13: Condition Key Species	Trend: Declining	Robust	High	Very High
Question 14: Human Activities and Living Resources	Status: Fair	Robust	High	Very High
Question 14: Human Activities and Living Resources	Trend: Declining	Robust	High	Very High
Maritime Archaeological Resources
Question 15: Integrity	Status: Fair	Not updated	Not updated	Not updated
Question 15: Integrity	Trend: Undetermined	Not updated	Not updated	Not updated
Question 16: Threat to Environment	Status: Fair	Medium	Medium	Medium
Question 16: Threat to Environment	Trend: Declining	Medium	Medium	Medium
Question 17: Human Activities	Status: Good/Fair	Not updated	Not updated	Not updated
Question 17: Human Activities	Trend: Undetermined	Not updated	Not updated	Not updated

Offshore Environment - Confidence Scoring Table

Question	2015 Rating	Evidence *(Limited, Medium or Robust)*	Agreement *(Low, Medium or High)*	Confidence *(Very Low, Low, Medium, High or Very High)*
Water Quality
Question 1: Multiple Stressors	Status: Fair	Medium	High	High
Question 1: Multiple Stressors	Trend: Declining	Robust	High	Very High
Question 2: Eutrophic Condition	Status: Good/Fair	Robust	High	Very High
Question 2: Eutrophic Condition	Trend: Declining	Medium	Medium	Medium
Question 3: Risks to Human Health	Status: Good/Fair	Not updated	Not updated	Not updated
Question 3: Risks to Human Health	Trend: Undetermined	Not updated	Not updated	Not updated
Question 4: Human Activities and Water Quality	Status: Fair	Not updated	Not updated	Not updated
Question 4: Human Activities and Water Quality	Trend: Improving	Not updated	Not updated	Not updated
Habitat
Question 5: Major Habitat	Status: Fair	Medium	High	High
Question 5: Major Habitat	Trend: Improving	Low	High	Medium
Question 6: Biologically-Structured	Status: Fair/Poor	Medium	High	High
Question 6: Biologically-Structured	Trend: Undetermined	Low	High	Medium
Question 7: Contaminants	Status: Fair	Medium	High	High
Question 7: Contaminants	Trend: Declining	Medium	High	High
Question 8: Human Activities and Habitat	Status: Fair	Medium	High	High
Question 8: Human Activities and Habitat	Trend: Improving	Medium	High	High
Living Resources
Question 9: Biodiversity	Status: Fair	Medium	Medium	Medium
Question 9: Biodiversity	Trend: Not changing	Low	Medium	Low
Question 11: Non-Indigenous Species	Status: Good	Not updated	Not updated	Not updated
Question 11: Non-Indigenous Species	Trend: Not changing	Not updated	Not updated	Not updated
Question 12: Status Key Species	Status: Good/Fair	Medium	Medium	Medium
Question 12: Status Key Species	Trend: Not changing	Medium	Low	Low
Question 13: Condition Key Species	Status: Good/Fair	Medium	Medium	Medium
Question 13: Condition Key Species	Trend: Declining	Medium	Low	Low
Question 14: Human Activities and Living Resources	Status: Fair	Medium	Medium	Medium
Question 14: Human Activities and Living Resources	Trend: Not changing	Medium	Medium	Medium
Maritime Archaeological Resources
Question 15: Integrity	Status: Undetermined	Not updated	Not updated	Not updated
Question 15: Integrity	Trend: Undetermined	Not updated	Not updated	Not updated
Question 16: Threat to Environment	Status: Fair	Medium	Medium	Medium
Question 16: Threat to Environment	Trend: Declining	Medium	Medium	Medium
Question 17: Human Activities	Status: Good/Fair	Not updated	Not updated	Not updated
Question 17: Human Activities	Trend: Undetermined	Not updated	Not updated	Not updated

Seamount Environment - Confidence Scoring Table

Question	2015 Rating	Evidence *(Limited, Medium or Robust)*	Agreement *(Low, Medium or High)*	Confidence *(Very Low, Low, Medium, High or Very High)*
Water Quality
Question 1: Multiple Stressors	Status: Undetermined	N/A	N/A	N/A
Question 1: Multiple Stressors	Trend: Undetermined	N/A	N/A	N/A
Question 2: Eutrophic Condition	Status: Undetermined	N/A	N/A	N/A
Question 2: Eutrophic Condition	Trend: Undetermined	N/A	N/A	N/A
Question 3: Risks to Human Health	Status: Undetermined	N/A	N/A	N/A
Question 3: Risks to Human Health	Trend: Undetermined	N/A	N/A	N/A
Question 4: Human Activities and Water Quality	Status: Good/Fair	Limited	High	Medium
Question 4: Human Activities and Water Quality	Trend: Undetermined	Limited	High	Medium
Habitat
Question 5: Major Habitat	Status: Good	Robust	High	Very High
Question 5: Major Habitat	Trend: Stable	Medium	High	High
Question 6: Biologically-Structured	Status: Good	Robust	High	Very High
Question 6: Biologically-Structured	Trend: Undetermined	Limited	High	Medium
Question 7: Contaminants	Status: Undetermined	N/A	N/A	N/A
Question 7: Contaminants	Trend: Undetermined	N/A	N/A	N/A
Question 8: Human Activities and Habitat	Status: Good/Fair	Medium	High	High
Question 8: Human Activities and Habitat	Trend: Undetermined	Limited	High	Medium
Living Resources
Question 9: Biodiversity	Status: Good	Robust	High	Very High
Question 9: Biodiversity	Trend: Undetermined	Medium	High	High
Question 11: Non-Indigenous Species	Status: Good	Limited	High	Medium
Question 11: Non-Indigenous Species	Trend: Not changing	Limited	High	Medium
Question 12: Status Key Species	Status: Good/Fair	Robust	Medium	High
Question 12: Status Key Species	Trend: Increasing	Medium	High	High
Question 13: Condition Key Species	Status: Good	Medium	High	High
Question 13: Condition Key Species	Trend: Not changing	Limited	High	Medium
Question 14: Human Activities and Living Resources	Status: Good/Fair	Medium	High	High
Question 14: Human Activities and Living Resources	Trend: Undetermined	Limited	High	Medium
Maritime Archaeological Resources
Question 15: Integrity	Status: N/A	N/A	N/A	N/A
Question 15: Integrity	Trend: N/A	N/A	N/A	N/A
Question 16: Threat to Environment	Status: N/A	N/A	N/A	N/A
Question 16: Threat to Environment	Trend: N/A	N/A	N/A	N/A
Question 17: Human Activities	Status: N/A	N/A	N/A	N/A
Question 17: Human Activities	Trend: N/A	N/A	N/A	N/A

Appendix B: Consultation with Experts and Document Review Monterey Bay National Marine Sanctuary

Estuarine Environment - Confidence Scoring Table

Nearshore Environment - Confidence Scoring Table

Offshore Environment - Confidence Scoring Table

Seamount Environment - Confidence Scoring Table

Appendix B: Consultation with Experts and Document Review
Monterey Bay National Marine Sanctuary