Use the DataONE API to automatically update the Network page

[meg3mach]

Member node community counts from dataone.org/network does not match search.dataone.org

For example, OPC is listed as upcoming on the community page, but shows in search as a member.
CERP is not listed on the community page, but is listed on search as a member.
Count of 44 repositories on the community page but 46 on search.

Appears to be an updating issue on the community dataone.org/network page.

[mbjones]

Great catch. We have several new members listed in Onboarding that are not in the web list. We've also had a couple drop, and we need to decide how to represent those -- several still have data in the network even though they are no longer connected via repositories. For example, GLEON was a member node, but now their data are in EDI -- so as a project they are still putting their data in DataONE, but they've changed "how". I'd like to represent that broader organizational participation wherever possible. A few repos fit this category.

[robyngit]

I've updated the preview version of the website with changed I'm working on to update the member repository list using the DataONE list nodes API, and I'm looking for some feedback. You can preview the changes here: https://preview.dataone.org/network/#section__current-member-repositories

I've filtered out coordinating nodes and categorized member nodes by operational status. Two deprecated notes and one new node are excluded from the list for the reasons listed below (click to expand for details):

DRP - no operationalStatus

{
    "identifier": "urn:node:DRP",
    "name": "Defense Resiliency Platform Against Extreme Cold Weather",
    "description": "The Defense Resiliency Platform (DRP) is a research-based multidisciplinary cyberinfrastructure platform enabling data integration and formatting, visualization, predictive analytics, and automated mapping of Arctic terrain data with the goal of maximizing the performance of the U.S. Army/CRREL in extreme cold weather environments.",
    "baseURL": "https://drp.dataone.org/metacat/d1/mn",
    "subjects": [
        "CN=urn:node:DRP,DC=dataone,DC=org"
    ],
    "contactSubjects": [
        "http://orcid.org/0000-0002-6014-8942"
    ],
    "replicate": false,
    "synchronize": false,
    "type": "mn",
    "state": "up",
    "synchronization": {
        "schedule": "10 0/3 * * * ? *",
        "lastHarvested": "2022-08-03T00:33:39.248Z",
        "lastCompleteHarvest": "1900-01-01T00:00:00.000Z"
    },
    "metacatVersion": "2.19.0",
    "upgradeStatus": "success",
    "readOnlyMode": false
}

EDORA - operationalStatus deprecated

{
    "identifier": "urn:node:EDORA",
    "name": "Environmental Data for the Oak Ridge Area (EDORA)",
    "description": "Environmental Data for the Oak Ridge Area (EDORA), including the Walker Branch Watershed.  These projects have all contributed to a more complete understanding of how forest watersheds function and have provided insights into the solution of energy-related problems associated with air pollution, contaminant transport, and forest nutrient dynamics. This is one of a few sites in the world characterized by long-term, intensive environmental studies.",
    "baseURL": "http://mercury-ops2.ornl.gov/EDORA_MN/mn",
    "subjects": [
        "CN=urn:node:EDORA,DC=dataone,DC=org"
    ],
    "contactSubjects": [
        "CN=Jim Green A10401,O=Google,C=US,DC=cilogon,DC=org"
    ],
    "replicate": false,
    "synchronize": true,
    "type": "mn",
    "state": "down",
    "synchronization": {
        "schedule": "0 0,5,10,15,20,25,30,35,40,45,50,55 * * * ? *",
        "lastHarvested": "2014-09-24T13:49:47.000Z",
        "lastCompleteHarvest": "1900-01-01T00:00:00.000Z"
    },
    "infoURL": "http://mercury-ops2.ornl.gov/edora/",
    "dateDeprecated": "2019-03-05T00:00:00.000Z",
    "dateUpcoming": "2014-05-29T00:00:00.000Z",
    "logoURL": "https://raw.githubusercontent.com/DataONEorg/member-node-info/master/production/graphics/web/EDORA.png",
    "operationalStatus": "deprecated",
    "dateOperational": "2014-11-24T00:00:00.000Z",
    "location": "-84.3228202,35.9346854"
}

RGD - operationalStatus deprecated

{
    "identifier": "urn:node:RGD",
    "name": "Regional and Global biogeochemical dynamics Data (RGD)",
    "description": "Regional and global biogeochemical dynamics data can be used to improve our understanding of the structure and function of various ecosystems; to enable prediction across spatial and temporal scales; and to parameterize and validate terrestrial ecosystem models.",
    "baseURL": "http://mercury-ops2.ornl.gov/RGD_MN/mn",
    "subjects": [
        "CN=urn:node:RGD,DC=dataone,DC=org"
    ],
    "contactSubjects": [
        "CN=Jim Green A10401,O=Google,C=US,DC=cilogon,DC=org"
    ],
    "replicate": false,
    "synchronize": true,
    "type": "mn",
    "state": "down",
    "synchronization": {
        "schedule": "0 0,5,10,15,20,25,30,35,40,45,50,55 * * * ? *",
        "lastHarvested": "2014-09-25T12:35:52.000Z",
        "lastCompleteHarvest": "1900-01-01T00:00:00.000Z"
    },
    "infoURL": "http://daac.ornl.gov/mercury.html",
    "dateDeprecated": "2019-03-05T00:00:00.000Z",
    "dateUpcoming": "2014-05-29T00:00:00.000Z",
    "logoURL": "https://raw.githubusercontent.com/DataONEorg/member-node-info/master/production/graphics/web/RGD.png",
    "operationalStatus": "deprecated",
    "dateOperational": "2014-11-24T00:00:00.000Z",
    "location": "-84.3228202,35.9346854"
}

Questions:

• Do we want to change how we organize repositories?
• How can we detect if a repository is upcoming? (dateUpcoming is in the future?)
• Should we display deprecated nodes?
• If nodes are missing information like DRP currently, should we display them, and in which category?

[robyngit]

Answers & tasks from our meeting today:

• Do we want to change how we organize repositories?

• Hide status for contributing repositories (done_
  (Also hid status for upcoming repositories, when there are some)

• How can we detect if a repository is upcoming? (dateUpcoming is in the future?)

• operationalStatus will be set to upcoming
• Updated the site to use this property to organize any upcoming repos based on this status

• If nodes are missing information like DRP currently, should we display them, and in which category?

• No, keep them filtered out

• Should we display deprecated nodes?

• To discuss further. Perhaps we can show them as "contributing"

[robyngit]

These changes have been merged into the main branch now, but we will continue discussion on how we should handle the deprecated nodes that currently exist and others in the future.