🔷 [ProjectTracking] Consolidate gathering node telemetry data

[khorolets]

Goals

Gain ownership of the nodes' telemetry data with Near One.

Currently, the telemetry gathering endpoint and logic behind it are implemented under near-explorer, which is officially sunset and will not be continued soon.

The database containing telemetry data is created on the same instance as ExplorerDB. The latter is constantly under stress, which is the main reason for the NEAR Explorer project's deprecation.

We at Near One / Protocol require telemetry data for different reasons, and it's not up to me to tell you that. Currently, there is no official owner of that data and its pipeline, causing much trouble during issues or outages.

Example: @gmilescu contacted me because the mainnet telemetry database is filled with data from statelessnet. The root cause of the problem is that the nodes from the statelessnet have https://explorer.mainnet.near.org/api/nodes by default in the config.json. Since a change of the config parameter on the decentralized node that we don't run is barely possible, we cannot solve this problem.

Links to external documentations and discussions

N/A

Estimated effort

0.25-0.5 HC for the technical part (steps: 1, 2, 4, 5) and who knows how much time for the "politics" part (steps: 3, 6

1. Spin up the PostgreSQL database under Near One "jurisdiction" for the telemetry data
2. Define/implement the simple endpoint for telemetry to be pushed (it might be a standalone service or even an entire NEAR Explorer if necessary)
3. Work with Pagoda's SRE team to proxy the existing endpoints to our service from step 2
4. (optional) Change the default config.json we publish to have our new endpoint
5. (optional) Change the default value for config.json we generate running neard init
6. Bridge the access to the telemetry data owned by us to Pagoda

Assumptions

This initiative assumes that the telemetry data is crucial for Near One. Otherwise, it doesn't make sense to invest in it.

Pre-requisites

• Near One GCP access
• Dedicated domain(s) for the telemetry service
• (preferable) Near One Grafana to set pg-connector to visualize the telemetry data on dashboards

Out of scope

• NEAR Explorer's ExplorerDB
• NEAR Explorer itself and Validators page of it