1 Executive Summary

1.1 This report arises from the work of an ad hoc working group on data and information systems and services (DISS) set up at the 4^th IGOS Partners meeting in Stockholm, Sweden in November 1999. The report was prepared as the result of a 3 day meeting from the 26^th to the 28^th April 2000 hosted and organized by NASDA and RESTEC in Tokyo. The topic of DISS has not yet been considered in the work of the Integrated Global Observation Strategy's Partnership (IGOS-P), yet clearly if its work is to be successful it is essential that products are made available in suitable forms and in a timely fashion to users. As in all IGOS Partnership activities the intention is not to propose new international structures or facilities unless absolutely necessary, but to propose how existing capabilities can be used and, if needed, modified to satisfy requirements.

1.2 The Working Group adopted a strategic viewpoint in its considerations and dealt with seven main topics: Its report first proposes a number of DISS principles that should be adopted by the IGOS Partnership (section 3). Second it proposes how considerations of DISS should be integrated into the IGOS Theme process (section 4) and then outlines some of the necessary steps in ensuring the improved generation of products (section 5) and how it is possible to improve knowledge and awareness of new products (section 6). In terms of networking and access the focus in this report is on the role of high performance research and educational networks as a contribution to the adoption and implementation of operational networks (section 7). The issue of capacity building and DISS is considered in section 8. Finally the contributions of existing international organizations to DISS issues is considered (section 9). A consolidated list of recommendations is provided in section 10.

2 Introduction

2.1 Because of the importance of Data and Information Systems and Services (DISS) IGOS arguably needs a strategy to deal with them. Such a strategy should be minimalist in nature, not involving any activities which are better performed at other levels.

2.2 The need for a DISS strategy arises because:

- The useful outputs from IGOS will almost always products rather than the observations themselves.

- Observations and products useful for the detection of medium to long term change require special efforts in terms of data processing to ensure that they are internally consistent both temporally and spatially.

- Calibration, validation, and quality assessment normally require an integrated approach involving producers and users of observations.

- Global products are often reliant on a cooperation between multiple providers of observations that have to be processed in an integrated fashion.

2.3 Following a presentation of a paper outlining some of the issues relating to IGOS and DISS an Ad Hoc Working Group was set up by the 4^th IGOS Partners' meeting in Stockholm (November 1999), under the chairmanship of Dr. John Townshend. Members of the Partnership were invited to participate in its activities.

2.4 Taking account of the recommendations made at the 4^th IGOS Partners meeting in Stockholm (November 1999), the major objectives of the Ad Hoc Working Group were to do the following:

- Consider DISS issues at a broad strategic level.

- Explore the extent to which the different policies of the Partners would allow the development of any common approaches.

- Consider and develop where possible "best practices" rather than "principles".

- Prepare a preliminary report for the 5^th Meeting of the IGOS Partnership.

3 Principles associated with DISS

3.1 It is recommended that the IGOS Partners adopt a set of DISS principles stated below and, in line with the priority the IGOS Partnership has set for itself, that these principles be applied to all IGOS-P's implementation activities. These DISS principles are of a sufficiently general nature as to allow those responsible for individual activities, such as Theme Teams, to complement or interpret them in ways appropriate to the specific activity. More detailed guidelines for the application of these principles follow in Section 4.

3.2 In submitting this recommendation, it is recognized that most IGOS Partners have a data policy¹ tailored to their individual needs and approved by their governing bodies. Similarly, within CEOS, each space agency has a data policy relating to data and data use against which it conducts its Earth Observation activities. Those individual data policies, sometimes legally binding, are often the result of long discussion and negotiation, and each is designed for purposes distinct from IGOS-P. It is not recommended, therefore, that the IGOS Partnership attempt to design an umbrella data policy.

3.3 It is recommended that each Partner ensure that its own data policy can be applied with sufficient flexibility to enable commitments made under the aegis of the IGOS Partnership be fulfilled. It is noted that, in developing their individual data policies, the IGOS Partners have all recognized the specific needs of the science and research communities and it is important in the furtherance of the IGOS that this specificity continue to be taken into account.

3.4 Accordingly, the following are recommended as DISS principles for the IGOS-Partnership. It is also proposed that each approved IGOS Partnership activity should be required to explain in its reports how it proposes to adhere to each of these principles. The principles should apply both to long-term and short-term observation activities.

1) Those responsible for observations need to ensure sufficient long-term records.

- Commitment to the provision of long-term observations by responsible agencies should be based on clearly articulated user needs.

- Calibration procedures should be implemented so that observations can be related to objective external standards.

- Validation of higher products needs to be carried out to ensure long-term reliability and consistency.

2) Full and open sharing of the full suite of data sets and products for all climate and environmental research and monitoring is a fundamental objective.

3) Information must be collected, analyzed, and distributed so that the veracity and consistency of the record can be assessed.

- Routine and permanent mechanisms need to be put in place to evaluate and monitor observing system performance, including calibration and metadata (including documentation of observing system characteristics).

- Periodic information on monitoring of data quality, including assessment of random errors and long-term systematic biases, must be assembled and made available for distribution.

- This information should be objectively assessed by the wider community.

- Assembly and wide distribution of all information and analyses must be carried out in the assessment of the long-term record.

- Periodic information regarding calibration and validation, metadata, and procedures used to process the data must be distributed widely.

4) Where there are potential disturbances to the record (e.g., due to changing technology), there must be a commitment to ensuring that the internal consistency of the record is maintained.

- Studies must be carried out to assess the impacts of new technologies as they affect the record (prior to and during implementation).

- The extent to which products, their applications and the objectives of users are vulnerable to potential disturbances should be assessed.

- These studies should involve the wider user community.

5) Archives must include easily accessible information about the data holdings, including quality assessments, supporting ancillary information, and guidance and aids for locating and obtaining the data.

6) International and where appropriate suitable national standards should be used to the greatest extent possible for media and for processing and communication of data sets.

3.5 The above principles have been developed against the background of guidance provided at various times by several international bodies, such as the U.N Framework Convention on Climate Change Guidelines for National Reporting, UN Principles Relating to Remote Sensing of the Earth from Outer Space, and the Data Policy for the International Geosphere-Biosphere Programme. In an appendix to this report the GCOS/GOOS/GTOS Climate Monitoring Principles adopted by the 5th session of the Conference of the Parties of the FCCC are provided.²

4 Incorporating Data Information Systems and Services into IGOS Partnership Themes

Rationale for Incorporating DISS Into IGOS Partnership Theme Teams

4.1 It is recommended that Theme Teams use the following guidelines when planning the development and implementation of new efforts or activities. These guidelines are an extension of the overarching basic strategic-level DISS principles set forth earlier in this document. Guidelines provide a road-map to ensure comprehensive consideration of key elements of DISS prior to and during the implementation phases of themes presented to and endorsed by the IGOS Partnership.

4.2 Information Technology is now in the hands of very diverse user groups and the capability to create products and services from data is rapidly expanding. Therefore, the need to design into observing systems the ability and capacity to collect large volumes of data, deliver, process, provide access, generate products and provide long-term stewardship are paramount. Specifically, the IGOS Partnership provides a forum for a strategic perspective on the end-to-end consideration of all DISS components of data and information collection, access, and utilization to the benefit of the widest spectrum of user types (e.g., environmental, climate change, socio-economic, policy/decision makers, the public).

Guidelines for incorporating DISS Activities

4.3 The following are the guidelines that should be adopted in Theme Team activities:

1. To the maximum extent possible, identify and utilize existing telecommunications networks to deliver data, products and services to a world-wide user community, as well as from the observing system, to the data collection, processing, product generation, and archival activities, i.e. the end-to-end telecommunication process.

Consideration should be paid to the use of available, existing and emerging telecommunication technologies for remote areas, as well as areas of little infrastructure.

2 The agency sponsoring the observing system is responsible for providing adequate metadata (i.e. data about data ) for the life cycle of the system's operation. These include levels of metadata beyond just directory and inventory information.

Metadata help organize, maintain, and insure the investment in data. Metadata can describe many different aspects of data to include but not limited to:

- Identification (Name of the data and data set, Who is responsible for the data, Geographic coverage, Themes of information included in the data, Currency of the data, and Restrictions on access and use of the data);

- Description of how the observations were collected, such as instruments, history of instruments and site/platform (space and in situ), calibration methods and coefficients, maintenance history, etc.;

- Data Quality which describes how good are the data including mention of any Quality Control processing used to identify any changes to original observational data, as well as any retrospective reprocessing;

- Spatial Data Organization and Reference model/system used;

- Data Formats;

- Where data are stored and how to obtain the products (location, types of delivery media, tape/ftp, and the costs).

Metadata should, to the maximum extent possible, utilize accepted formats and standards (e.g. FGDC, ISO TC-211, CIP).

Promote the interoperability of systems' metadata even in cases where the metadata does not conform to an existing standard.

3. Identify the potential site(s) that will be assigned the responsibility of providing the short and/or long-term storage and access of these observational data and metadata. Maximum use of existing data centers (e.g. the World Data Center system - see section 9) is strongly encouraged. The locations of data inventories and catalogs, as well as products, should be well identified. Products for distribution need not necessarily be co-located and stored at the same location as the data, data sets, and metadata.

4. Identify the potential locations, other than data centers, where products and services may be available for access by the user community.

5. Identify potential classes of users for the data, products, and services and the potential benefits to all IGOS Partnership themes (e.g. environmental, social, and economic).

6. Rapid and convenient access to data, products, and services should be considered in the design of the theme (e.g. required high speed/wide bandwidth access to data and products on disk and/or automated robotic storage systems).

7. Identify any potential products that can be rapidly generated and distributed using the initial observations, before processing and quality control at a data center.

8. Data formats should, to the maximum extent possible, be developed for compatibility with data fusion and visualization programs and techniques (e.g. GIS and data modeling) for utility in more efficient products and services generation.

The need for data mining of large array data bases and data sets (e.g. those from EOS, ENVISAT, ADEOS-2, METOP) should be a consideration in data format and structure.

Long-term logical data storage formats should be independent of hardware systems and specific software formats (e.g. use of ASCII flat files is preferred to use of BUFR/GRIB formats).

9. Toolkits for the ingest and visualization of the data and products should be considered in the design of the system.

10. Demonstration and prototype systems should take into account the potential for the transition to operational systems and plan accordingly using these guidelines (e.g. system life cycle considerations).

11. Disposition schedules for data, products, and services should be developed to define the period of retention.

12. Evaluation of the impacts to the users (e.g. maintaining the continuity of data, products, and services) due to potential changes in instrumentation and other processing techniques need to be factored into the planning of new systems.

13. A final review process prior to disposition should be identified by the provider to preclude the possible loss of valuable data and their products.

14. Evaluation of the impacts to the users (e.g. maintaining the continuity of data, products, and services) due to potential changes in instrumentation and other processing techniques need to be factored into the planning of new systems.

15. Define any restrictions or limitations to data, products, and services access and distribution consistent with national policies relative to free and open access.

Level of Commitment Required for Providing DISS into Themes and Projects

4.4 Theme Teams should use this structure as a basis for periodic reporting on the progress and status of their efforts. It is further recommended that each Theme Team include a person with knowledge and expertise in Data Information Systems and Services. Individuals at Data Centers can be one source of this expertise.

5 Improved generation of products.

5.1 Key tangible results of the progress and success of IGOS are the Earth science products generated by the implementation activities of the Partners. A further indicator of success (or failure) of IGOS is the degree to which this new suite of Earth science products are available to, and used by, the world-wide community of scientists, resource managers, policy makers, and others. Since the success or failure of the IGOS Partnership depends on the ability of the partners to generate products and to make them readily available, the Data and Information Systems and Services (DISS) of the IGOS Partners must be sufficiently coordinated, flexible, and capable of providing support for a wide variety of IGOS themes and other implementation activities.

5.2 An important premise underlying the operation of DISS functions in the activities of IGOS Partnership is that the various system and service contributions of the Partners must have considerable freedom in the ways in which they implement their functions and capabilities. Implementation will not be centrally developed, nor will the pieces developed be centrally managed. However, all DISS functions should be configured in such a way that data and information of one part may be accessible to all other DISS components that need it. As outlined below, this aspect of DISS functionality can be enabled primarily via standards and practices that are the province of existing bodies specifically concerned with such activities.

5.3 While it is not possible at this time to define specific products that will be generated under the direction of IGOS Themes, it is clear, however, that significant volumes of observational data must be integrated from a wide variety of Partners remote sensing missions, and that the DISS functions of the IGOS Partners must be capable of handling such data volumes. It is equally apparent that it must also be possible to handle, integrate and distribute data from in situ observations and model outputs. As described below, the responsibility for the specific definition of IGOS data products usually begins with the scientific community via principal investigators acting individually or in groups.

Agreements on standards and protocols.

5.4 DISS functions of IGOS activities will consist of highly distributed, heterogeneous components of the Partners. These widely varying elements will be responsible for enabling product generation. Because the IGOS Partners have already made considerable investments in DISS activities, it is clear that the IGOS Partnership will leverage off of existing systems and services. The future structure of DISS functions could be quite different, however, as new contributions are added and as existing components take advantage of technological innovations and otherwise evolve to meet future science, environmental, and economic demands.

5.5 Given that the elements of DISS functions for IGOS Partnership activities comprise a variety of interdependent components, a key goal will be to harmonize and aggregate these various, disparate, and perhaps numerous elements. Adherence to data system standards and protocols is one important means for reaching harmony between these disparate system elements.

5.6 The definition of standards and protocols for archiving and access to Earth science data is currently the province of existing bodies, such as CEOS, the world wide web consortium, ISO, and others. These standards bodies use a set of defined processes to select, maintain, and evolve their interface standards.

5.7 It is therefore recommended that the IGOS Partners should not try to duplicate or replicate these DISS efforts for the specific purposes of the IGOS-P. Rather, the Partners should promote and should encourage the work of appropriate standards bodies in support of IGOS-Partnership objectives and to use accepted standards and formats to the maximum extent possible.

Definition of key IGOS Partnership sets of products

5.8 A key goal of DISS activities is to harmonize and aggregate the various, disparate, and numerous data and information systems and services of the Partners and to harness these capabilities for the generation of useful products. The user community must not just be passive recipients of the data, but should be actively involved in the definition, validation and generation of the data.

5.9 Products arising from the work of the IGOS Partnership will usually originate in the imagination and invention of investigators working individually and in teams. In practice, these investigators may reside in academia, government laboratories, in NGOs, and in commercial for-profit companies. Regardless of their provenance, it is clearly beyond the scope of the Partnership as a whole to define all IGOS products. Many products will arise from individual and collective scientific efforts in response to the Themes.

5.10 While the IGOS Partnership will not specify the contents or algorithms for deriving science data products, the Partnership must support the means for enabling product generation. Consideration by Partners should given to the support of regional science networks and to the archiving and access of data products generated by the specific activities initiated under IGOS Themes.

5.11 It is clear that IGOS partnership can not directly manage the efforts of investigators in generating data products, nor could the Partnership directly manage the activities of regional science networks. However, the fundamental importance of the products, and the science networks in which they are incubated, archived, and distributed is also clear. Therefore, there is an obvious need for the IGOS Partnership to establish some form of reporting and feedback between the individuals and organizations that generate products and services and the Strategy that spawned them.

5.12 The IGOS Partnership is directed towards an end-to-end process from observations through assimilation, product generation, and product use. As part of a continuing process to assess the success of the IGOS Partnership, it will be necessary to ensure that commitments made to generate products are maintained and that the resultant products are used on a sustained basis by those users, who committed to utilize them.

5.13 It is recommended that procedures be set up allowing an assessment of the success of the IGOS Partnership activities by monitoring the consistency of product generation and the uptake of products by users who committed to sustained use of the data. Examples of the benefits derived from the data and products should be part of the assessment process.

6 Targeting New User Groups and Improving Knowledge about Data Sets and Products

Targeting new user groups

6.1 The IGOS Partnership might occasionally decide to target a particular type of user as happened when the secretariats of international conventions were approached as a group, but normally new user groups will come through expressions of interest in one or more Themes. Each Theme Team report should address the steps needed to widen the user community.

6.2 Special attention should continue be given to fostering a good interface with the secretariats of international conventions. If one or more Partners can provide an effective link to such a convention secretariat they should be invited to represent IGOS-P's interest in this respect.

6.3 The IGOS Partnership needs to publicize its work widely and encourage Partners to do likewise since they are often nearer to the user interface.

6.4 The best way of increasing use is to tackle the problems raised in the following section.

Improving knowledge about data sets and products

6.5 The key to increased use is the effective diffusion of information about the availability of existing (or soon to be available) data sets and products.

6.6 Each Theme Team should be asked to ensure that the availability and conditions of products is adequately advertised and forms an important part of the Theme Team report. This may entail making recommendations for providing a point of contact even after the Theme Team will have been disbanded.

6.7 The IGOS Partnership should not create a new mechanism for announcing data or product availability, but should seek to obtain help and advice of expert groups assisted by one or more Partners.

6.8 Although there is still pressure from users for a single point of access for space data, at least for the near future, inquiries should be directed to the agency providing the data.

7 Networking and improving access

7.1 A fundamental element of the infrastructure needed for successful implementation of IGOS-Partnership Theme Team activities is the world-wide collection of internet protocol (IP) networks. For the purpose of this report, such IP networks can be categorized as follows:

- Private networks of the operational Partners. These networks are designed specifically for mission-critical operations and are characterized by a high degree of security and reliability. Consequently, severe restrictions are placed on access to these networks. While access to the private operational networks is necessarily restricted, data flow from the more restricted private network to less secure (but more readily available) networks is typically permitted.

- High-performance research and education networks (HPRENs). These networks are being implemented by the major developed countries to provide proof-of-concept development and demonstration of next generation internet technologies and applications. Use of these networks is generally restricted to "meritorious applications" that meet a predefined set of acceptable use policies (AUPs). While each HPREN has its AUP and procedures for approving access to and usage of network services, the HPRENs are typically targeted to promoting science, engineering and educational prototype applications that require high-performance network connectivity or services. By restricting access to the network to those applications that meet AUP criteria, the HPRENs can generally provide a high-level of network performance to those meritorious applications that are permitted to use the network. These networks are typically characterized by a level of security and performance reliability that is somewhat less restrictive than that of the private operational networks, but the performance and security of HPRENs are still considerably better than the world-wide internet.

- The world-wide internet. The world-wide internet has essentially no guarantees of performance, reliability or security built into its basic architecture. Other benefits of the internet clearly outweigh these concerns, however, and clear evidence of this fact is the internet's dramatic growth, accessibility, global reach and obvious utility in numerous applications. It is equally clear that the world-wide internet has its limits in Earth science applications that require even modest levels of data flow performance, reliability or security.

- Other networks. For the sake of completeness, it is useful to note that there are many other IP networks of less immediate interest to IGOS partners, including those of commercial Internet services providers and other private enterprises.

7.2 HPRENs are likely to be of particular importance to IGOS. As described above, these networks provide performance on the order of that expected to be needed by IGOS in the implementation of its Themes. Furthermore, it is expected that most, if not all, of the IGOS Partnership Themes could be defined as "meritorious applications" that meet the AUPs of the HPREN providers.

7.3 While there are obvious strengths in the HPRENs for IGOS, some obvious weaknesses are also apparent. For one, the current suite of HPRENs has been implemented predominantly in the industrialized nations of the northern hemisphere . Secondly, the points of presence (POP) for access to the HPRENs typically exist within a few major cities of the nations that support them, although this problem is less severe in the US, where the HPRENs are well integrated into universities and research organizations. Also, the cost of carrying HPREN services from a local POP to a participating organization or institution must be borne by that institution. It has often been the case that the most difficult aspect of bringing a trans-national and trans-oceanic HPREN service extending thousands of kilometers to a local institute has been in establishing the "local loop" from the institute to the POP, even if the local connection needs to extend 5 km or less. This "last mile" problem should be considered as one of the issues for capacity building associated with IGOS-P, especially in developing nations where the problem is most acute.

7.4 A final consideration of the HPRENs that must be taken into account is that the AUPs of these networks typically prohibit their use for operational activities, although some exceptions exist. In the event that an IGOS Partnership activity become fully operational, the networks supporting the project would most likely need to transition from a research and education network to an operational one. Obviously, sufficient resources would then need to be available to effect this transition.

7.5 It is recommended that the IGOS Partnership recognize the potential value of HPRENs as tools for IGOS Partnership Themes that meet the acceptable use policies of these networks. Where appropriate the IGOS Partners should encourage the expansion of existing national HPRENs and they should also encourage the expansion of HPREN capacity into additional nations and regions (especially in the southern hemisphere) to facilitate the full use of products arising from its activities. The IGOS Partnership should recognize the importance of the potential "last mile" problem for gaining access to existing HPRENs and seek to overcome this problem especially in developing nations.

7.6 The CEOS WGISS has recently begun to evaluate the availability and performance of HPRENs as a contribution to the Carbon Theme through its interactions with GOFC (see also Section 9 of this report).

8 Capacity Building and DISS within IGOS

8.1 The success of IGOS Partnership depends on the extent to which the products are generated globally and are successfully applied. Any limitations in capacity can significantly limit the range and extent of use of these products. Capacity should equally include an ability to provide a contribution in the form of in situ data.

8.2 The universality of IGOS Partnership often breaks down at a regional or national level, including in developed countries because of the lack of capacity to contribute data especially in situ data, to access and to utilize the products.

8.3 It is in the interest of the IGOS Partnership as a whole for Partners to be encouraged to deploy their own resources towards building capacity in critical areas. At the level of the IGOS Partnership there are a number of possible actions to be taken which would be complementary to those of the individual partners.

8.4 It is recommended that the IGOS Partnership pay particular attention to the importance of DISS issues in the context of capacity building. The following are of particular importance in the context of DISS activities:

- Efforts should be made by IGOS Partnership to enhance capacity for the development and dissemination of products rather than observations to users. In this context, the IGOS Partnership should facilitate mechanisms, for bringing together observations from variety of sources and packaging digital information in a way that add value to it and making available through standard formats as is done for example by UNEP/GRID.

- Enormous digital divides exist between many classes of users in terms of computing equipment, connectivity, and the ability to use them. There are major divides between developed and developing countries, but also between different types of users within countries, e.g. in many countries there is limited network capacity for educational establishments. The IGOS Partnership should consider working with international funding agencies towards bridging this gap.

- Products arising from activities of IGOS Partnership should be made available in forms appropriate for use at national and sub-national scales.

- Outreach activities to decision-makers are valuable. Developing case studies demonstrating the benefits of improved observations and products and the use of geo-spatial technologies applied to real life problems, in part as a response to requests from Earthwatch in the context of Agenda 21, provide such an example of the value of outreach efforts.

- Improvements in networking capabilities are important and members of the IGOS Partnership are encouraged to assist where they can towards such improvements especially in developing countries (see section 7).

9 The Contributions of Existing International Organizational Structures to DISS and IGOS-P

9.1 When a Theme Team is developing its activity, it is recommended that subject-matter expertise from a number of existing international data management organizations should be employed to aid the DISS portion of the Theme. The services and expertise available from some of these organizations include the following.

CEOS Working Group on Information Systems and Services (WGISS)

9.2 The overall objective of WGISS is to facilitate data and information management and services for users and data providers in dealing with global, regional and local issues. It addresses in particular the capture, description, processing, access, retrieval, utilization, maintenance and exchange of spaceborne Earth observation data and supporting ancillary and auxiliary data and information, enabling improved interoperability and interconnectivity of information systems and services. WGISS has three main sub-groups on Access, Data and Networking and for each of these a number of specific tasks as outlined below. The emphasis of all of these activities is primarily with respect to the remote sensing data derived from the sensor systems of CEOS members. However, the expertise and knowledge are transferable for use with in-situ systems. These subgroups and tasks are tabulated below.

9.3 The Ad Hoc Working Group noted, that at the recent meeting in London, the valuable interactions between WGISS and activities contributing to the Carbon Theme through its interactions with the Global Observations of Forest Cover Project (GOFC). WGISS is developing a Test Environment that will contribute to the IGOS Themes, by calling upon WGISS tools and services held and operated by CEOS WGISS members and associates. At the same time using this approach WGISS intends to develop and improve its tools and products to better serve users' needs. WGISS has the potential to contribute to the work of the Theme Teams and to assist in the subsequent implementation following the reporting of the Theme Team's recommendations.

Table 1 WGISS Activities

Access Sub-group's Tasks

International Directory Network (IDN)

World Wide Web (WWW)

CEOS Interoperability Extensions (CINTEX)

Data Sub-group's Tasks

Archive

Global Mapping and Visualization

Global Data Sets

Guidelines

Data Services

Networking Sub-group

CEOSNet

Network Resource Planning and Coordination

Network Performance Measurement

CEOS Working Group on Calibration and Validation

9.4 The need to generate internally consistent data sets provides a heavy premium on the need for reliable calibration and validation of remotely sensed data. It is therefore anticipated that the WGCV has the potential to assist the work of the Theme Teams in preparing their reports and to assist in the subsequent implementation of their recommendations, especially in those circumstances where the data comes from multiple sensors and from different CEOS members.

World Data Centers (WDC)

9.5 The World Data Center (WDC) System under the aegis of the International Council for Science has a well-established role in the long-term stewardship, provision of open access, and quality control of many key environmental data sets, including baseline data sets such as GHCN, CARDS and COADS. It provides historical descriptions and perspectives on the data sets and ensures that there is observational network monitoring and performance assessments. It also uses its products to provide regional and global assessments. It actively seeks to identify new potential products and services, as well as potential users of these products and services. Data sets held by WDC's include both in situ and remotely sensed data sets.

9.6 In the evolution of the IGOS Partnership there will be a need for Data Centers to act as stewards and to provide access to data, products and services. Data Centers can also contribute in a significant way to assisting with the development of the DISS related activities of the IGOS Partnership Theme Teams.

9.7 Given the WDC's long standing international role in these functions, we recommend that the IGOS Partnership consider their contributions in the work of the Theme Teams and in the subsequent implementation of their recommendations.

Joint Data Implementation Management Panel (JDIMP)

9.8 The Joint Data Implementation Management Panel was set up jointly by GCOS, GOOS, and GTOS, a precursor panel having been set up to serve the needs of GCOS. The remit of JDIMP includes in situ data sets, as well as remotely sensed data. Currently, the Panel does not appear to be functioning and clarification is sought from IGOS Partners as to its status and future.

Global Observing System Information Center (GOSIC) - University of Delaware

9.9 The Global Observing System Information Center (GOSIC) provides information on the observing requirements, the operational data systems and the access procedures for finding and obtaining data and products of the G3OS. The G3OS consists of the Global Climate Observing System (GCOS), the Global Ocean Observing System (GOOS), and the Global Terrestrial Observing System (GTOS). GOSIC includes information on the GCOS Surface Network (GSN), GCOS Upper Air Network (GUAN) and Global Atmospheric Watch (GAW).

9.10 GOSIC can be accessed using standard Web browsers, with gateways to a relational database for complex searches. GOSIC links directly to other sites on the Web that provide G3OS data and information. GOSIC encourages the participation of other sites in supporting the G3OS data and information systems and encourages the standardization of presentations and methodologies between these sites and GOSIC. Functional Elements of GOSIC include metadata, data sets, data visualization, program documentation and web links to like resources. It is anticipated that GOSIC could be very useful to IGOS DISS, and the Ad Hoc Working Group encourages close collaboration with it and continued support for GOSIC.

Other areas of expertise relating to DISS

9.11 There is much expertise in several of the other Partners relating to DISS. These include activities within IGBP, WMO, IOC, FAO and UNEP, which can contribute to the work of the Partnership and its Theme Teams.

Contributions of existing international structures to the work of the Partnership.

9.12 In the various Panels, committees and other organizations referred to above there is considerable expertise in the area of Data and Information System and Services, which readily could assist in the work of the Theme Teams. While we do not recommend that any formal panel or committee be set up drawn from representatives of these groups, there could be value for each organization, if willing, to nominate a contact person who could be consulted by Theme Teams during their work associated with DISS issues.

10 Recommendations

10.1 The following lists the recommendations made in the report. It also provides the main proposed DISS principles and the DISS guidelines recommended for all IGOS Themes. The paragraph in the report, where each recommendation is found, is indicated in parentheses.

Recommendation 1 It is recommended that the IGOS Partners adopt a set of DISS principles stated below and, in line with the priority the IGOS Partnership has set for itself, that these principles be applied to all IGOS-P's implementation activities (3.1).

1) Those responsible for observations need to ensure sufficient long-term records.

2) Full and open sharing of the full suite of data sets and products for all climate and environmental research and monitoring is a fundamental objective.

3) Information must be collected, analyzed, and distributed so that the veracity and consistency of the record can be assessed.

4) Where there are potential disturbances to the record there must be a commitment to ensuring that the internal consistency of the record is maintained.

5) Archives must include easily accessible information about the data holdings, including quality assessments, supporting ancillary information, and guidance and aids for locating and obtaining the data.

6) International and where appropriate suitable national standards should be used to the greatest extent possible for media and for processing and communication of data sets.

Recommendation 2 It is … proposed that each approved IGOS Partnership activity should be required to explain in its reports how it proposes to adhere to each of these principles. The principles should apply both to long-term and short-term observation activities (3.4).

Recommendation 3 It is not recommended … that the IGOS Partnership attempt to design an umbrella data policy (3.2).

Recommendation 4 It is recommended that each Partner ensure that its own data policy can be applied with sufficient flexibility to enable commitments made under the aegis of the IGOS Partnership be fulfilled. (3.3).

Recommendation 5 It is recommended that Theme Teams use the following guidelines when planning the development and implementation of new efforts or activities. These guidelines are an extension of the overarching basic strategic-level DISS principles set forth earlier in this document. Guidelines provide a road-map to ensure comprehensive consideration of key elements of DISS prior to and during the implementation phases of themes presented to and endorsed by the IGOS Partnership (4.1).

1. To the maximum extent possible, identify and utilize existing telecommunications networks to deliver data, products and services to a world-wide user community, as well as from the observing system, to the data collection, processing, product generation, and archival activities, i.e. the end-to-end telecommunication process.

2. The agency sponsoring the observing system is responsible for providing adequate metadata for the life cycle of the system's operation. These include levels of metadata beyond just directory and inventory information.

3. Identify the potential site(s) that will be assigned the responsibility of providing the short and/or long-term storage and access of these observational data and metadata. Maximum use of existing data centers is strongly encouraged. The locations of data inventories and catalogs, as well as products, should be well identified. Products for distribution need not necessarily be colocated and stored at the same location as the data, data sets, and metadata.

4. Identify the potential locations, other than data centers, where products and services may be available for access by the user community.

5. Identify potential classes of users for the data, products, and services and the potential benefits to all IGOS Partnership themes.

6. Rapid and convenient access to data, products, and services should be considered in the design of the theme.

7. Identify any potential products that can be rapidly generated and distributed using the initial observations, before processing and quality control at a data center.

8. Data formats should, to the maximum extent possible, be developed for compatibility with data fusion and visualization programs and techniques.

9. Toolkits for the ingest and visualization of the data and products should be considered in the design of the system.

10. Demonstration and prototype systems should take into account the potential for the transition to operational systems and plan accordingly using these guidelines.

11. Disposition schedules for data, products, and services should be developed to define the period of retention.

12. Evaluation of the impacts to the users due to potential changes in instrumentation and other processing techniques need to be factored into the planning of new systems.

13. Define any restrictions or limitations to data, products, and services access and distribution consistent with national policies relative to free and open access.

14. Evaluation of the impacts to the users, due to potential changes in instrumentation and other processing techniques need to be factored into the planning of new systems.

15. Define any restrictions or limitations to data, products, and services access and distribution consistent with national policies relative to free and open access.

Recommendation 6 Theme Teams should use this structure (from Recommendation 5) as a basis for periodic reporting on the progress and status of their efforts. It is further recommended that each Theme Team include a person with knowledge and expertise in Data Information Systems and Services. Individuals at Data Centers can be one source of this expertise (4.4).

Recommendation 7 It is ….. recommended that the IGOS Partners should not try to duplicate or replicate these DISS efforts for the specific purposes of the IGOS-P. Rather, the Partners should promote and should encourage the work of appropriate standards bodies in support of the IGOS Partnership objectives and to use accepted standards and formats to the maximum extent possible (5.7).

Recommendation 8 It is recommended that procedures be set up allowing an assessment of the success of IGOS Partnership activities by monitoring the consistency of product generation and the uptake of products by users who committed to sustained use of the data. Examples of the benefits derived from the data and products should be part of the assessment process (5.13).

Recommendation 9 It is recommended that the IGOS Partnership recognize the potential value of HPRENs as tools for the IGOS Partnership Themes that meet the acceptable use policies of these networks. Where appropriate the IGOS Partners should encourage the expansion of existing national HPRENs and they should also encourage the expansion of HPREN capacity into additional nations and regions especially in the southern hemisphere) to facilitate the full use of products arising from its activities. The IGOS Partnership should recognize the importance of the potential "last mile" problem for gaining access to existing HPRENs and seek to overcome this problem especially in developing nations (7.5).

Recommendation 10 It is recommended that the IGOS Partnership pay particular attention to the importance of DISS issues in the context of capacity building (8.4).

Recommendation 11 When a Theme Team is developing its activity, it is recommended that subject-matter expertise from a number of existing international data management organizations should be employed to aid the DISS portion of the Theme (9.1).

Recommendation 12 Given the WDC's long standing international role in these functions, we recommend that the IGOS Partnership consider their contributions in the work of the Theme Teams and in the subsequent implementation of their recommendations (9.7).

Participants at the Tokyo Meeting of the Ad Hoc Working Group on IGOS-P and DISS

Family Name	Organization	E-mail
Kohei ARAI	SAGA UNIVERSITY	arai@hijiri.jp.is.saga-u.ac.jp
Pierre BAUER	CNES DPI/OT	pierre.bauer@education.gouv.fr  pierre.bauer@cnes.fr
Howard DIAMOND	NOAA/NESDIS	howard.diamond@noaa.gov
Roy GIBSON	EUMETSAT	roy.gibson@wanadoo.fr
Yukio HARUYAMA	RESTEC	haruyama@restec.or.jp
Tadahiko INADA	NASDA	inada.tadahiko@nasda.go.jp
Chu ISHIDA	NASDA	ishida.chu@nasda.go.jp
Hiroshi ISHIGURO	RESTEC	ishiguro@restec.or.jp
John JENSEN	NOAA/NESDIS/NCDC	jjensen@ncdc.noaa.gov
Shinji KAMIJO	NASDA	Kamijo.shinji@nasda.go.jp
Gilbert KIRKHAM	NASA	kirkhamGR@state.gov
Kazuko MISAWA	RESTEC	misawa@restec.or.jp
Linda MOODIE	NOAA	Linda.Moodie@noaa.gov
Masami ONODA	NASDA	onoda.masami@nasda.go.jp
Matt SCHWALLER	NASA	matt@rattler.gsfc.nasa.gov
Ashbindu SINGH	UNEP/GRID-Sioux Falls	singh@edcmail.cr.usgs.gov
Shin-Ichi SOBUE	NASDA	sobue.shinichi@nasda.go.jp
Gerard SZEJWACH	IGBP-DISS	szejwach@pik-potsdam.de
John TOWNSHEND	UNIV. of MARYLAND	jt59@umail.umd.edu

GCOS/GOOS/GTOS CLIMATE MONITORING PRINCIPLES
adopted by the 5th Session of the Conference of the Parties of the FCCC.

CONFERENCE OF THE PARTIES
Fifth session
Bonn, 25 October - 5 November 1999
Agenda item 4 (h)

1. Assess the impact of new systems or changes to existing systems prior to implementation.

2. Require a suitable period of overlap for new and old observing systems.

3. Treat the results of calibration, validation, algorithm changes, and data homogeneity assessments with the same care as data.

4. Ensure a capability to routinely assess the quality and homogeneity, including high-resolution data and related descriptive information, of data on extreme events.

5. Integrate environmental climate-monitoring products and assessments, such as IPCC assessments, into global observing priorities.

6. Maintain uninterrupted stations and observing systems.

7. Give a high priority to additional observations in data-poor regions and regions sensitive to change.

8. Provide long-term requirements at the outset of new system design and implementation to network operators, designers and instrument engineers.

9. Promote the conversion of research observing systems to long-term operations in a carefully planned manner.

10. Data management systems that facilitate access, use and interpretation are essential

1 The term "data policy" commonly refers primarily to issues of ownership, terms and conditions of use and charging for an entity's data products.

2 Many of these policies and principles can be found in the publications of EOPOLE (Earth Observation Data Policy and Europe). Their web site is at http://www.geog.ucl.ac.uk/eopole.

3 GCOS-39 (WMO/TD-No.87) (UNEP/DEIA/MR.97-8) (GOOS-11) (GTOS-11) Report of the GCOS/GOOS/GTOS Panel, third session (Tokyo, Japan, 15-18 July 1997).

IGOS Homepage  Earthwatch Homepage