Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update install and publish process #131

Merged
merged 37 commits into from
Apr 9, 2024

Conversation

matthewkeil
Copy link
Member

Description

  • update .dockerignore and .gitignore
  • update workflow for new build and publish process and move to workflows folder to turn on and test
  • update Dockerfile used to build linux/arm64 bindings
  • fix tsconfigs to only support cjs for the moment
  • remove index.mjs to only support cjs for now
  • update index.js and index.d.ts to support new build process and for dynamically finding bindings
  • remove dependency on node-fetch and move jazzer.js to optional dependencies to support linux/arm64 build
  • add install.js script that sources bindings and prepares them for runtime use (finds in prebuilt bundle, github release or builds locally)
  • add scripts/makeRelease to move all CI prebuilds to release folder and then selectively move back the bundled version to the npm published prebuild folder
  • add check to fuzz tests to ensure optional dep is installed and throws if not
  • turn on unit test for randomBytesNonZero

@matthewkeil matthewkeil requested a review from a team as a code owner April 8, 2024 06:39
Copy link

github-actions bot commented Apr 8, 2024

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 1349653 Previous: - Ratio
PublicKey serialization 1.1990 us/op
PublicKey deserialize 22.397 us/op
PublicKey deserialize and validate - 1 keys 83.747 us/op
PublicKey deserialize and validate - 100 keys 8.2852 ms/op
PublicKey deserialize and validate - 10000 keys 835.01 ms/op
SecretKey.fromKeygen 3.5030 us/op
SecretKey serialization 1.0580 us/op
SecretKey deserialization 1.4600 us/op
SecretKey.toPublicKey 138.18 us/op
SecretKey.sign 531.22 us/op
Signature serialization 1.2820 us/op
Signature deserialize 44.097 us/op
Signatures deserialize and validate - 1 sets 120.25 us/op
Signatures deserialize and validate - 100 sets 11.949 ms/op
Signatures deserialize and validate - 10000 sets 1.1973 s/op
aggregatePublicKeys - 1 sets 2.5090 us/op
aggregatePublicKeys - 8 sets 10.550 us/op
aggregatePublicKeys - 32 sets 37.880 us/op
aggregatePublicKeys - 128 sets 147.05 us/op
aggregatePublicKeys - 256 sets 294.35 us/op
aggregateSignatures - 1 sets 4.2670 us/op
aggregateSignatures - 8 sets 23.951 us/op
aggregateSignatures - 32 sets 91.120 us/op
aggregateSignatures - 128 sets 361.36 us/op
aggregateSignatures - 256 sets 724.08 us/op
aggregateVerify - 1 sets 1.5808 ms/op
aggregateVerify - 8 sets 5.7655 ms/op
aggregateVerify - 32 sets 20.433 ms/op
aggregateVerify - 128 sets 79.293 ms/op
aggregateVerify - 256 sets 157.39 ms/op
verifyMultipleAggregateSignatures - 1 sets 1.6359 ms/op
verifyMultipleAggregateSignatures - 8 sets 6.1635 ms/op
verifyMultipleAggregateSignatures - 32 sets 22.059 ms/op
verifyMultipleAggregateSignatures - 128 sets 85.720 ms/op
verifyMultipleAggregateSignatures - 256 sets 170.54 ms/op
Same message - 1 sets 1.7120 ms/op
Same message - 8 sets 2.5781 ms/op
Same message - 32 sets 5.5421 ms/op
Same message - 128 sets 17.318 ms/op
Same message - 256 sets 33.056 ms/op
libuv multithreading - addVerificationRandomness true 20.676 s/op
libuv multithreading - addVerificationRandomness false 20.656 s/op

by benchmarkbot/action

Dockerfile Outdated Show resolved Hide resolved
binding.gyp Outdated
@@ -1,63 +1,70 @@
{
'targets': [
"targets": [
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the changes in this file were to get Windows to publish. It’s mostly changes from to .

The only other meaningful changes are noted below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated. I reverted all the quote changes. The windows build error was coming from somewhere else. I noticed an indentation inconsistency in the file so when I was in there editing I reverted the quotes to the more standard '.

"<!@(node -p \"require('node-addon-api').include_dir\")",
"include_dirs": [
"blst/bindings",
"<!@(node -p \"require('node-addon-api').include\")",
Copy link
Member Author

@matthewkeil matthewkeil Apr 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a known issue with how windows builds native modules. Need to use the include export instead of include_dir for it to be fully cross-compatible.

binding.gyp Outdated
'AdditionalOptions': [ '/std:c++17' ],
"conditions": [
[ "OS!='win'", {
"sources": ["blst/build/assembly.S"],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This source is not for windows so refactored a bit to move it to a conditional.

"node-gyp": "^10.0.1"
}
"optionalDependencies": {
"@jazzer.js/core": "^2.1.0"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lib is not compatible with linux-arm64. I attempted to fix it but just ended up fighting a battle that was not worth it. Its an issue with how the base C lib exports artifacts for CMake in @jazzer/fuzzer. I put up an issue in their repo here CodeIntelligenceTesting/jazzer.js#795

Dockerfile Show resolved Hide resolved
.github/workflows/main.yml Outdated Show resolved Hide resolved
async function buildBindings(binaryName: string): Promise<string> {
await exec("npm run clean:gyp", true, {cwd: ROOT_DIR});
await exec("npm run build:gyp", true, {cwd: ROOT_DIR});
const bindingPath = getBindingsPath(ROOT_DIR);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after we use node-gyp do we know the exact location? looks like this getBindingsPath function is more suitable for other places not this one because it involves a search

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the times yes but not always.

Because its possible to npm install --debug which will create a debug build in build/Debug instead of in build/Release. So its necessary to check several places...

scripts/makeRelease.ts Outdated Show resolved Hide resolved
},
"main": "./dist/lib/index.js",
"types": "./dist/lib/index.d.ts",
"files": [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we also specified this in the publish workflow:

- name: Create Github release with prebuilds
        if: steps.tag.outputs.tagcreated == 'yes'
        uses: softprops/action-gh-release@v2
        with:
          files: release/*
          tag_name: ${{ steps.tag.outputs.tagname }}
          fail_on_unmatched_files: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

do they serve different purposes? looks like releases/* only contains the binding files there

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The releases are for the github release and the prebuilds get included in the npm bundle. I added information explaining this in CONTRIBUTING.md

.github/workflows/main.yml Outdated Show resolved Hide resolved
@matthewkeil matthewkeil merged commit 692d201 into master Apr 9, 2024
22 checks passed
@matthewkeil matthewkeil deleted the mkeil/migrate-rebuild-update-install-and-publish branch April 22, 2024 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants