add Global Replay (#143)

guide/sections/part2/global-services.adoc

@@ -9,6 +9,7 @@ Depending on the nature of the Global Service, the following is the minimum capa
 * Three (3) Global Caches: Each Global Cache is connected to at least two (2) Global Brokers and should be able to download the data from all WIS2 Nodes providing core data
 * Two (2) Global Discovery Catalogues: Each Global Discovery Catalogue is connected to at least one (1) Global Broker
 * Two (2) Global Monitors: Each Global Monitor should scrape the metrics from all other Global Services
+* One (1) Global Replay: Each Global Replay connected to at least one (1) Global Broker
 
 In addition to the above, WIS architecture can accommodate adding (or removing) Global Services. Candidate WIS centres should inform their WIS National Focal Point and contact the WMO Secretariat to discuss their offer to provide a Global Service.
 
@@ -165,6 +166,7 @@ In WIS2 Global Caches provide access to WMO core data for data consumers. This a
 ** Searchable Catalog - Filtering (Deployment)
 ** JSON (Building Block)
 ** HTML (Building Block)
+* A Global Discovery Catalogue shall subscribe to the topic ``++origin/a/wis2/+/metadata/#++``.
 * The Global Discovery Catalogue will make discovery metadata available via the collection identifier of `wis2-discovery-metadata`.
 * The Global Discovery Catalogue advertises the availability of datasets and how to access them or subscribe to updates.
 * The Global Discovery Catalogue does not advertise or list the availability of individual data objects that comprise a dataset (that is, data files).
@@ -183,6 +185,7 @@ In WIS2 Global Caches provide access to WMO core data for data consumers. This a
 * A Global Discovery Catalogue will generate and store a zip file of all WCMP2 records once a day, that will be made be accessible via HTTP.
 * A Global Discovery Catalogue will publish a WIS2 Notification Message of its zip file of all WCMP2 records on its centre-id's +metadata+ topic (for example, `origin/a/wis2/centre-id/metadata`, where `centre-id` is the centre identifier of the Global Discovery Catalogue).
 * A Global Discovery Catalogue may initialize itself (cold start) from a zip file of all WCMP2 records published.
+* A Global Discovery Catalogue may query a Global Replay for metadata messages published and process those messages to insert/update/delete WCMP2 records, for downtimes of less than 24 hours.
 * As a convention Global Discovery Catalogue centre-id will be ``tld-{centre-name}-global-discovery-catalogue``.
 
 ===== 2.7.5.2 Global Discovery Catalogue reference implementation: wis2-gdc
@@ -215,3 +218,38 @@ The wis2-gdc is managed as a free and open source project.  Source code, issue t
 * As a convention Global Monitor centre-id will be ``tld-{centre-name}-global-monitor``.
 
  The main task of the Global Monitor is to regularly query the provided metrics from the relevant WIS2 entities, aggregate and process the data and then provide the results to the end user in a suitable presentation.
+
+==== Global Replay
+
+===== Technical considerations
+
+* The Global Replay provides Global Services and Data Consumers with a mechanism to search and query for notification messages of interest.
+* The Global Discovery Catalogue implements the OGC API – Features – Part 1: Core standardfootnote:[OGC-API Features - Part 1 TODO ADD LINK], adhering to the following conform    ance classes and their dependencies.
+** TODO: add Requirements classes
+** JSON (Building Block)
+* A Global Replay shall subscribe to the topics `+origin/a/wis2/#+` and `+cache/a/wis2/#+`.
+* The Global Replay will make notification messages available via the collectio identifier of `wis2-notification-messages`.
+* A single Global Replay instance is sufficient for WIS2.
+* Multiple Global Replay instances may be deployed for resilience.
+* Global Replay instances operate independently of each other; each Global Replay instance will hold notification messages according to the required retention period.  Global Replays do not need to synchronise between themselves.
+* A Global Replay is populated with notification messages from a Global Broker instance.
+* A Global Replay should connect and subscribe to more than one Global Broker instance to ensure that no messages are lost in the event of a Global Broker failure. A Global Replay instance will discard duplicate messages as needed.
+* A Global Replay will validate notification messages against the WIS Notification Message (WNM).  Valid WIS Notification Messages will be ingested into the index.  Invalid or malformed notification messages will be discarded. TODO should we validate WNM? or moot point?validate
+* A Global Replay will add a property called ``properties.topic`` to identify the topic from the which the notification message was published.
+* A Global Replay will remove notification messages after the required retention period.
+* As a convention Global Replay centre-id will be ``tld-{centre-name}-global-replay``.
+
+===== Global Replay reference implementation: wis2-grep
+
+To provide a Global Replay, members may use whichever software components they consider most appropriate to comply with WIS2 Technical Regulations.
+
+To assist Members participation in WIS2, a free and open-source Global Replay Reference Implementation is made available for download and use.  wis2-grep builds on mature and robust free and open-source software components that are widely adopted for operational use.
+
+wis2-grep provides functionality required for the Global Replay, providing the following technical functions:
+
+* notification messages subscription and publication from the Global Broker
+* notification message validation, ingest and publication (TODO: should we validate or not?)
+* WNM compliance (TODO: should we validate or not?)
+* OGC API - Features - Part 1: Core compliance
+
+wis2-grep is managed as a free and open source project.  Source code, issue tracking and discussions are hosted in the open on GitHub: https://github.com/wmo-im/wis2-grep.

guide/sections/part2/wis2-architecture.adoc

@@ -42,6 +42,7 @@ These roles are outlined below.
   ii) Global Broker: provides highly available messaging services where users may subscribe to notifications about all datasets provided by data publishers.
   iii) Global Cache: provides highly available download service for cached copies of core data downloaded from data publishers’ web-services.
   iv) Global Monitor: gathers and displays system performance, data availability, and other metrics from all WIS2 Nodes and Global Services.
+  v) Global Replay: provides access to WIS Notification Messages via a searchable and queryable API.
 
 ==== 2.2.4 Data consumer
 * This role represents anyone wanting to find, access, and use data from WIS2 – examples include (but are not limited to): NMHS, Government agencies, research institutions, private sector organizations, and so on.
@@ -68,7 +69,7 @@ Leveraging existing open standards, WIS2 defines the following specifications in
 
 |WIS2 Notification Message
 |Dataset metadata, dataset granules
-|Global Broker, WIS2 Nodes
+|Global Broker, Global Replay, WIS2 Nodes
 
 |===
 
@@ -121,6 +122,13 @@ Please refer to the _Manual on WIS_, Volume II for details.
   ii) Whether data can be effectively accessed by data consumers.
   iii) The performance of components in the WIS2 system.
 
+==== 2.4.6 Global Replay
+* WIS2 may include a Global Replay.
+* A Global Replay enables Global Services and data consumers to search and query notification messages published by the Global Broker.
+* A Global Replay subscribes to notification messages via a Global Broker about the availability of new notification messages. It downloads a copy of the notification message and updates its local index.
+* A Global Replay shall retain copies of notification messages for a duration compatible with the real-time or near real-time schedule of the data and not less than 24-hours.
+* A Global Replay will delete notification messages from its local index once the retention period has expired.
+
 === 2.5 Protocols configuration
 
 ==== 2.5.1 Publish-subscribe protocol (MQTT)

guide/sections/part2/wis2node.adoc

@@ -115,4 +115,4 @@ The wis2box provides functionality required for both data publisher and data con
 
 Project documentation can be found at https://docs.wis2box.wis.wmo.int.
 
-The wis2box is managed as a free and open source project.  Source code, issue tracking and discussions are hosted openly on GitHub: https://docs.wis2box.wis.wmo.int.
+The wis2box is managed as a free and open source project.  Source code, issue tracking and discussions are hosted openly on GitHub: https://docs.wis2box.wis.wmo.int.