Repository navigation
v8-canary builds missing since May 17, linux/darwin builds missing since Febrary #312
Description
Activity
Currently the main issue is the Rust version requirement
We should be getting Linux builds now (I updated Rust on the RHEL 8 release machines (all archs)).
I'm suspicious that 20260517 is the last available downloadable build as we have in staging, e.g
total 644852 drwxrwxr-x 3 staging staging 4096 May 21 16:42 . drwxrwxr-x 1968 staging staging 131072 May 21 11:14 .. drwxrwxr-x 3 staging staging 4096 May 21 16:15 docs -rw-rw-r-- 1 staging staging 0 May 21 16:15 docs.done -rw-rw-r-- 1 staging staging 84122889 May 21 11:47 node-v27.0.0-v8-canary20260521da8e25c394-aix-ppc64.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 11:42 node-v27.0.0-v8-canary20260521da8e25c394-aix-ppc64.tar.gz.done -rw-rw-r-- 1 staging staging 10088167 May 21 16:15 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 16:16 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.gz.done -rw-rw-r-- 1 staging staging 596044 May 21 16:16 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.xz -rw-rw-r-- 1 staging staging 0 May 21 16:16 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.xz.done -rw-rw-r-- 1 staging staging 59259661 May 21 14:46 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 14:48 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.gz.done -rw-rw-r-- 1 staging staging 31185060 May 21 14:48 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.xz -rw-rw-r-- 1 staging staging 0 May 21 14:48 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.xz.done -rw-rw-r-- 1 staging staging 62065750 May 21 11:50 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 11:52 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.gz.done -rw-rw-r-- 1 staging staging 31804332 May 21 11:52 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.xz -rw-rw-r-- 1 staging staging 0 May 21 11:53 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.xz.done -rw-rw-r-- 1 staging staging 61479246 May 21 11:12 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 11:15 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.gz.done -rw-rw-r-- 1 staging staging 31380780 May 21 11:14 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.xz -rw-rw-r-- 1 staging staging 0 May 21 11:15 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.xz.done -rw-rw-r-- 1 staging staging 59602003 May 21 16:39 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 16:42 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.gz.done -rw-rw-r-- 1 staging staging 32253548 May 21 16:42 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.xz -rw-rw-r-- 1 staging staging 0 May 21 16:42 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.xz.done -rw-rw-r-- 1 staging staging 131238376 May 21 16:02 node-v27.0.0-v8-canary20260521da8e25c394.tar.gz -rw-rw-r-- 1 staging staging 0 May 21 16:11 node-v27.0.0-v8-canary20260521da8e25c394.tar.gz.done -rw-rw-r-- 1 staging staging 65089448 May 21 16:11 node-v27.0.0-v8-canary20260521da8e25c394.tar.xz -rw-rw-r-- 1 staging staging 0 May 21 16:12 node-v27.0.0-v8-canary20260521da8e25c394.tar.xz.donewhich suggests something is going wrong with promotion for the canary builds.
This is the cause:
dist@infra-digitalocean-ubuntu2204-x64-2:~$ /home/staging/tools/promote/promote_nightly.sh nodejs Promoting v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3/node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.7z... rm: cannot remove '/home/dist/nodejs/test/v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3/node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.7z': Permission denied dist@infra-digitalocean-ubuntu2204-x64-2:~$File cannot be removed because the target directory is old enough that it was made write only for the
distuser:total 93936 drwxr-xr-x 3 root root 4096 Apr 22 20:30 . drwxr-xr-x 208 dist dist 20480 May 4 12:30 .. -rw-r--r-- 1 root root 24080359 Apr 22 20:00 node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.7z -rw-r--r-- 1 root root 38229424 Apr 22 20:00 node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.zip -rw-r--r-- 1 root root 33837056 Apr 22 20:00 node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-x64.msi -rw-r--r-- 1 root root 742 Apr 22 20:30 SHASUMS256.txt drwxr-xr-x 2 root root 4096 Apr 22 20:00 win-x64I believe the culprit is https://ci-release.nodejs.org/job/iojs+release-mefi-pgo/4/ (cc @StefanStojanovic ) which hasn't set a datestring field so it ends up reusing the target directory and promotion breaks because it cannot overwrite the now restricted directory. To fix this I'm going to removed the staged directory for
v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3so that the promote script doesn't try to overwrite the existing files.https://nodejs.org/download/v8-canary/v27.0.0-v8-canary20260521da8e25c394/ now exists with AIX and Linux builds. As @targos mentioned, the other missing platforms will need a newer version of Rust.
Reacted by Joyee Cheung@richardlau yes, it is my fault, and I apologize. One question: in the
datestringdescription, it says Only needed for nightly, and I was using test builds. For future builds of this job (I plan to run some today), what should I do to prevent the issue I caused? Just adddatestring?I'll start another build soon. I'll put a
datestringto avoid the issues described here.@richardlau yes, it is my fault, and I apologize. One question: in the
datestringdescription, it says Only needed for nightly, and I was using test builds. For future builds of this job (I plan to run some today), what should I do to prevent the issue I caused? Just adddatestring?Yes, setting the datestring would be the simplest way.
Looking at current ci-release v8-canary jobs, the only recent failing platform is Windows.
11:37:55 CUSTOMBUILD : Torque error : Cannot open file path/uri: src/builtins/wasm-to-js.tq"src/ebug/debug-wasm-objects.tq [c:\ws\tools\v8_gypfiles\run_torque.vcxproj] 11:37:55 C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for '..\..\out\Release\\torque.exe' exited with code 3. [c:\ws\tools\v8_gypfiles\run_torque.vcxproj]which looks like a quoting/delimiter issue?
However it also does look like https://nodejs.org/download/v8-canary/ is behind what is on the DO droplet.
- https://nodejs.org/download/v8-canary/v27.0.0-v8-canary20260904e6cba8a70b/ looks to be the latest version available
- The DO droplet has
v27.0.0-v8-canary20260918239c3aa75c(build is in progress so more platforms will appear as they complete).
@nodejs/web-infra I checked and it looks like we do have
v27.0.0-v8-canary20260918239c3aa75cin R2's dist so this looks like another caching issue with https://github.com/nodejs/release-cloudflare-worker/ similar to the nightlies (nodejs/release-cloudflare-worker#1018).Looking at current ci-release v8-canary jobs, the only recent failing platform is Windows.
11:37:55 CUSTOMBUILD : Torque error : Cannot open file path/uri: src/builtins/wasm-to-js.tq"src/ebug/debug-wasm-objects.tq [c:\ws\tools\v8_gypfiles\run_torque.vcxproj] 11:37:55 C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for '..\..\out\Release\\torque.exe' exited with code 3. [c:\ws\tools\v8_gypfiles\run_torque.vcxproj]which looks like a quoting/delimiter issue?
Above was from vs2022_clang-x64. Failure on vs202_clang-arm64:
11:17:48 CUSTOMBUILD : Torque error : Cannot open file path/uri: src/builtins/wasm-to-j.tq [c:\ws\tools\v8_gypfiles\run_torque_host.vcxproj] 11:17:48 C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for '..\..\out\Release\\torque_host.exe' exited with code 3. [c:\ws\tools\v8_gypfiles\run_torque_host.vcxproj]- The DO droplet has
v27.0.0-v8-canary20260918239c3aa75c(build is in progress so more platforms will appear as they complete).
@nodejs/web-infra I checked and it looks like we do have
v27.0.0-v8-canary20260918239c3aa75cin R2's dist so this looks like another caching issue with https://github.com/nodejs/release-cloudflare-worker/ similar to the nightlies (nodejs/release-cloudflare-worker#1018).How recent is this? There is caching in place, so I would not expect a new canary to be available instantly?
- The DO droplet has
- The DO droplet has
v27.0.0-v8-canary20260918239c3aa75c(build is in progress so more platforms will appear as they complete).
@nodejs/web-infra I checked and it looks like we do have
v27.0.0-v8-canary20260918239c3aa75cin R2's dist so this looks like another caching issue with https://github.com/nodejs/release-cloudflare-worker/ similar to the nightlies (nodejs/release-cloudflare-worker#1018).How recent is this? There is caching in place, so I would not expect a new canary to be available instantly?
That's fair, but then all the builds between 20260904 to today are missing on https://nodejs.org/download/v8-canary/ but are present on the DO droplet and R2 dist. i.e.
- v27.0.0-v8-canary202609153f670614ec
- v27.0.0-v8-canary20260916045505e938
- v27.0.0-v8-canary20260917d5e7bad223
we didn't have any builds between 4 September and 15 September. The directory cache for the worker was manually refreshed on 7 September.
- The DO droplet has
🤔 https://github.com/nodejs/release-cloudflare-worker/actions/workflows/update-links.yml runs every day and is meant to update the directory cache. @nodejs/web-infra I'm not sure I know enough about how the worker works to know why this cache isn't refreshing? Does the canary directory need to be added to https://github.com/nodejs/release-cloudflare-worker/blob/1bb1f5403a333564dde7d99dcc6cb75a01697a4a/scripts/constants.mjs#L61-L65 perhaps?
The Windows build should be (at least partially) fixed by nodejs/node@8273447
We can either wait for V8 15.2 to land on
mainor cherry-pick that fix tocanary-base(I don't have the permission for that)
Last one is AIX-only https://nodejs.org/download/v8-canary/v27.0.0-v8-canary202605175b9294c0c1/
I don't have access to release CI, so can't see what's wrong.
Also there has not been any linux/darwin builds since Feburary. Last one is https://nodejs.org/download/v8-canary/v26.0.0-v8-canary20260223fd2a99f348/
cc @nodejs/build