build!: optimize for x86-64-v3 cpu microarchitecture (Haswell+)

[lklimek]

Issue being fixed or feature implemented

In v1.6.2, we reverted change introducing CPU performance optimizations for x86-64 architecture.
We want these optimizations to be enabled in final release image.

We will require CPUs to have microarchitecture level x86-64-v3 (Haswell and newer).

We will still keep unoptimized code in debug images (the ones with -debug suffix) as a fallback mechanism.

What was done?

Enabled CPU optimizations for release images in:

• Rust
• C and C++ (CFLAGS, CXXFLAGS)
• RocksDB (PORTABLE flag)

How Has This Been Tested?

Started drive-abci inside userspace QEMU emulation

Test scenario

Release: test that should fail on usupported x86-64 CPUs

If this test fails, it means your CPU is not supported.
This test should always succeed for all architectures different than x86-64 (I mean arm).

docker run --rm -ti dashpay/drive:1.7.0-pr.2374.1 verify ; echo $?

As a result:

• passed - you should see the number 0 at the end of output
• failed - you should see number 132 at the end of output

Debug: test that should always pass

This test should always pass.

docker run --rm -ti dashpay/drive:1.7.0-pr.2374.1-debug verify ; echo $?

As a result, you should see the version and number 0 at the end of output

Results

x86-64-v1 (KVM model: kvm64)

• debug image (no optimizations): works
• release image: fails with Illegal instruction

x86-64-v2 (KVM model: Nehalem)

• debug: works
• release: fails with Illegal instruction

x86-64-v3 (KVM model: EPYC)

• debug: works
• release: works

Breaking Changes

Dash Platform nodes must be compatible with x86-64-v3 ABI compatibility level 3(x86-64-v3). It translates into Intel Haswell CPU and newer.

Nice, but not normative ,table to help select a CPU: https://www.qemu.org/docs/master/system/i386/cpu.html#abi-compatibility-levels-for-cpu-models

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
• I have made corresponding changes to the documentation if needed

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

Summary by CodeRabbit

• New Features

  • Introduced a new build argument in the Dockerfile for specifying the build profile.
  • Added a new target configuration for x86_64-unknown-linux-gnu with updated compilation flags.

• Bug Fixes

  • Updated Rust compilation flags for the x86_64-unknown-linux-musl target to enhance performance.

• Documentation

  • Updated package version to "1.7.0" and added new lints configuration for tokio_unstable.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This pull request introduces configuration updates for Rust compilation targeting x86_64 architectures across multiple files. The changes focus on enhancing build configurations in .cargo/config.toml and .cargo/config-release.toml by adding target-specific CPU optimization flags. The Dockerfile is modified to support a new CARGO_BUILD_PROFILE argument, providing more flexibility in build processes. Additionally, the drive-abci package version is bumped to 1.7.0, and a new lints configuration is added to handle unexpected configurations.

Changes

File	Change Summary
.cargo/config-release.toml	Updated rustflags for x86_64-unknown-linux-musl with -C target-cpu=x86-64-v3; Added new target configuration for x86_64-unknown-linux-gnu with similar flags
.cargo/config.toml	Updated rustflags for x86_64-unknown-linux-musl with -C target-cpu=x86-64; Added new target configuration for x86_64-unknown-linux-gnu
Dockerfile	Introduced CARGO_BUILD_PROFILE build argument (default: dev); Enhanced build logic for environment variables and build profiles
packages/rs-drive-abci/Cargo.toml	Bumped package version to 1.7.0; Added [lints.rust] section for unexpected configurations

Possibly related PRs

• ci: prebuilt librocksdb in docker image #2318: Modifications to Dockerfile for RocksDB static library build process
• ci: docker image always released as -dev #2358: Updates to Dockerfile for specifying build profiles

Suggested reviewers

• QuantumExplorer

Poem

🐰 Rustacean Rabbit's Build Ballad 🚀

Flags flying high, CPU optimized tight,
Configs dancing with compile-time might,
Docker's profile, a flexible friend,
Version bumped, our journey transcend,
Code leaps forward with computational glee! 🌟

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[qwizzie]

According ChatGPT the microarchitecture level x86-64-v3 will actually exclude Haswell CPU's and apply to Skylake or higher :

Source : https://chatgpt.com/share/675c6a2b-3dd0-8004-93db-7f4f643f724b

Update :

A new chat session with ChatGPT shows this Haswell CPU (and Haswell CPU's in general) after a misleading chat start does conform to x86-64-v3 :

Source : https://chatgpt.com/share/675d3e7c-7380-8004-babb-5f859d67d76f

To check in Ubuntu : lscpu | grep -o -E 'avx|avx2|bmi1|bmi2|fma|movbe|lzcnt|popcnt' | sort | uniq -c

So x86-64-v3 does seem to support my cpu after all and GLIBC-wise there is no problem either (i am on GLIBC 2.35). But maybe best to let me test this pull request merged in a future dev build on my VPS ? Just to be sure it will actually work on Haswell CPU's.

Update : my VPS provider also confirmed to me that my CPU conforms to X86-64-v3 and added this little piece of information :

We've check our E5-2698v3 and all flags are present, we also checked the qemu version and our qemu version is 8.1.5 which also clear the requirement of Qemu version being greater than 7.2

So its not just the flags that need to be in there and the Haswell microarchitecture of the CPU, the qemu version seems to play a role as well (needs to be greater than 7.2)

[lklimek]

@coderabbitai review all

[coderabbitai]

@lklimek, I will review all the changes in the PR as per your request.

✅ Actions performed

Full review triggered.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

Dockerfile (1)

280-283: Remove duplicate ONBUILD ARG CARGO_BUILD_PROFILE declaration

The ONBUILD ARG CARGO_BUILD_PROFILE=dev is declared twice (lines 117-118 and 282-283). This duplication is unnecessary and can be removed to simplify the Dockerfile.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b265bb8 and 4800738.

📒 Files selected for processing (4)

• .cargo/config-release.toml (1 hunks)
• .cargo/config.toml (1 hunks)
• Dockerfile (3 hunks)
• packages/rs-drive-abci/Cargo.toml (1 hunks)

🧰 Additional context used

📓 Learnings (1)

packages/rs-drive-abci/Cargo.toml (1)

Learnt from: shumkov
PR: dashpay/platform#2375
File: packages/rs-drive-abci/Cargo.toml:61-63
Timestamp: 2024-12-05T09:29:38.918Z
Learning: In the `drive-abci` package, avoid adding unused dependencies like `hashbrown` to `Cargo.toml`. The team relies on CI to detect dependency version issues.

🔇 Additional comments (5)

Dockerfile (1)

125-138: Verify CPU compatibility for optimization flags

The compiler flags -march=x86-64-v3 and -march=x86-64 set in the environment variables CFLAGS and CXXFLAGS optimize the build for specific CPU architectures. Ensure that all target deployment environments support these architectures to prevent runtime errors on unsupported CPUs.

Run the following script to list all occurrences where these flags are used and verify compatibility:

✅ Verification successful

Let me gather more context about the build configuration and architecture-specific settings.

---

Based on the gathered context, I can now provide a final response:

CPU optimization flags are properly configured with architecture safeguards

The optimization flags are safely implemented with proper architecture checks:

• For AMD64 release builds: Uses -march=x86-64-v3 for modern CPU optimizations
• For AMD64 debug builds: Uses conservative -march=x86-64 baseline
• For non-AMD64 architectures (e.g., ARM64): No architecture-specific flags are set (PORTABLE=1)

The Dockerfile includes proper architecture detection and fallbacks, making it safe across different platforms. The build system correctly handles both AMD64 and ARM64 architectures with appropriate conditional logic.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Find all Dockerfiles and check for usage of -march flags
fd Dockerfile | xargs grep -n '\-march='

Length of output: 292

---

Script:

#!/bin/bash
# Check for any architecture-specific conditions and build profiles
rg "TARGETARCH|CARGO_BUILD_PROFILE" -A 3 -B 3

# Check for any documentation or comments about CPU requirements
fd "README|INSTALL|CONTRIBUTING" -e md | xargs rg -i "cpu|processor|architecture|requirement"

Length of output: 6330

.cargo/config-release.toml (1)

8-11: Ensure consistency of rustflags across targets

The addition of -C target-cpu=x86-64-v3 to the rustflags for both x86_64-unknown-linux-musl and x86_64-unknown-linux-gnu targets is appropriate for optimizing performance on newer CPUs (Intel Haswell and newer). Ensure that this aligns with the project's hardware compatibility requirements.

.cargo/config.toml (2)

7-15: Update rustflags for x86_64-unknown-linux-musl target

The addition of -C target-cpu=x86-64 in the rustflags enhances performance while maintaining compatibility with standard x86-64 architectures in debug builds.

---

16-24: Add configuration for x86_64-unknown-linux-gnu target

Including rustflags for the x86_64-unknown-linux-gnu target ensures consistent build settings across similar architectures.

packages/rs-drive-abci/Cargo.toml (1)

117-118: Review the addition of the [lints.rust] section

The new [lints.rust] configuration helps in catching unintended usage of cfg(tokio_unstable) by issuing warnings for unexpected conditional compilation flags. This can improve code quality by alerting developers to potential misuse.

Note: Based on previous learnings from shumkov, the team prefers to minimize additions to Cargo.toml unless necessary. Since this configuration enhances code safety without introducing new dependencies, it aligns well with project practices.

[lklimek]

So x86-64-v3 does seem to support my cpu after all and GLIBC-wise there is no problem either (i am on GLIBC 2.35). But maybe best to let me test this pull request merged in a future dev build on my VPS ? Just to be sure it will actually work on Haswell CPU's.

@qwizzie I've updated the PR description with info on how to test these changes yourself. Any feedback appreciated :).

[qwizzie]

docker run --rm -ti dashpay/drive:1.7.0-pr.2374.1 verify ; echo $?

That works fine, thanks. It passes (0) on my VPS, but does not show version :

It also shows 0 with the debug test and passes without any error. But i am not seeing any version in the debug test.

[lklimek]

docker run --rm -ti dashpay/drive:1.7.0-pr.2374.1 verify ; echo $?

That works fine, thanks. It passes (0) on my VPS, but does not show version :

It's fine, my mistake (I changed the script but not description), will update. Thanks