refactor to more generic code

[MoLow]

Ref: nodejs/release-cloudflare-worker#33

in order to get Cloudflare to link these soft links, I've refactored the code to be more generic, this way we can use in CF workers:

const { S3Client, ListObjectsV2Command } = require('@aws-sdk/client-s3')
const { fromIni } = require('@aws-sdk/credential-providers')
const { Linker } = require('nodejs-latest-linker/common.js')

const ENDPOINT = 'https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com'
const PROFILE = 'worker'
const BUCKET = 'dist-prod'
const RELEASE_DIR = 'nodejs/release/'
const DOCS_DIR = 'nodejs/docs/'

const client = new S3Client({
  endpoint: ENDPOINT,
  region: 'auto',
  credentials: fromIni({ profile: PROFILE })
});

(async function main () {
  const allDirs = await listDirectory(RELEASE_DIR)
  const linker = new Linker({ baseDir: RELEASE_DIR, docsDir: DOCS_DIR })
  const links = await linker.getLinks(allDirs, dir => listDirectory(`${dir}/`))
  console.log(links)
})()

async function listDirectory (dir) {
  let truncated = true
  let continuationToken
  let items = []
  while (truncated) {
    const data = await client.send(new ListObjectsV2Command({ Bucket: BUCKET, Delimiter: '/', Prefix: dir, ContinuationToken: continuationToken }))
    items = items.concat(data.CommonPrefixes ?? []).concat(data.Contents ?? [])
    truncated = data.IsTruncated
    continuationToken = data.NextContinuationToken
  }
  return items.map((d) => d.Prefix ?? d.Key)
}

[MoLow]

CC @nodejs/build @nodejs/releasers

[MoLow]

Refs: nodejs/release-cloudflare-worker#34
Refs: nodejs/release-cloudflare-worker#36

[richardlau]

We should keep testing on Node.js 12 until we can upgrade the version of Node.js used to run this tool on our server.

[MoLow]

well, that breaks using modern javascript async/await ect :/
What is needed to upgrade node.js on the www server?

[richardlau]

Current server is running Ubuntu 16.04 and using nodesource to install Node.js. I'm not sure whether Nodesource provide later versions of Node.js on Ubuntu 16.04. I'd like to update the OS on the server but it's risky given how dependent we are on that droplet at the moment.

[MoLow]

so we would either need to adjust this to run on node 12 or upgrade node without using nodesource, since upgrading the OS is not trivial
I prefer the second option, is that possible?

[richardlau]

I think Node.js 18 is a non-starter on Ubuntu 16.04 due to glibc versions. Node.js 16 is a possibility -- there may even be a Nodesource distribution for it (I'd suggest testing in a separate Ubuntu 16.04 system first).

[richardlau]

@MoLow Yes, it should be.

[targos]

Based on the fact that https://deb.nodesource.com/node_16.x/dists/xenial/Release returns something, I think v16.x can be installed using the NodeSource distribution.

[MoLow]

I was able to install node16 on ubuntu 16.04 using nodesource. let me check these changes work on node 16

[MoLow]

tests have passed. @richardlau should I go ahead and update www nodejs version?

[richardlau]

@MoLow Yes (and keep an eye on tomorrow's nightly to check the indexer still runs ok). If you could also update https://github.com/nodejs/build/blob/b8b63ade1e76d8ccf669335ee384ed2ee34a3abe/ansible/www-standalone/tasks/base.yaml#L1-L3 (which says Node.js 8 but we are definitely running with 12 on the server) that would be great.

[MoLow]

I'v updated the www server to node 16. how does this package get published to npm?

[richardlau]

I'v updated the www server to node 16. how does this package get published to npm?

ah I'll need to add you as a maintainer in npm. I've just invited you via npm's web ui -- if it needs to be done via the CLI (we had some issues adding someone to citgm in npm) I'll have to do that tomorrow.

[MoLow]

Promotion and indexer worked as expected:
https://r2.nodejs.org/download/nightly/v21.0.0-nightly202310039c683204db/
https://r2.nodejs.org/download/nightly/index.json

vs
https://nodejs.org/download/nightly/v21.0.0-nightly202310039c683204db/
https://nodejs.org/download/nightly/index.json

CC @richardlau