Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker-targets v2 #10346

Merged
merged 12 commits into from
May 6, 2024
Merged

Docker-targets v2 #10346

merged 12 commits into from
May 6, 2024

Conversation

bitwiseguy
Copy link
Contributor

@bitwiseguy bitwiseguy commented Apr 30, 2024

Description

Reviving #8721. This PR was generated by:

  1. creating new branch off of docker-targets-v2 called ss/docker-targets-v2
  2. rebasing on top of develop branch
  3. applying updates to docker build flow for new components (i.e. op-dispute-mon, da-server) introduced since docker-targets-v2 was created

This PR accomplishes the following:

🟢 Adds a script to Circle CI config.yml to pass git tags to docker buildx so that docker containers report the appropriate git tag version of code that is used to build the image. Previously versions were always incorrectly reported as v0.0.0

🟢 Reduces the average duration of the op-*-docker-build Circle CI jobs reduced from ~4m45s --> ~45s (due to new docker build flow)

🟢 Adds a Circle CI job called check-cross-platform to ensure images can be run on ARM machines (even though the docker build+push is performed on an AMD machine)

Tests

Manually tested the new versioning script at the following commits to ensure the version was set correctly in each case:

  • rc tag: caf41c5
  • full release tag: 24a8d3e06e61c7a8938dfb7a591345a437036381
  • untagged commit: b036ae364125169e8b1d7915aecdcbac859cfdbe

Tested the new Circle CI job check-cross-platform by explicitly passing it the docker image tag e87e5ef2b96893eb8b446da420f7ba7f3e3c5985 for op-node. When this job is actually triggered (after docker-build is run with the publish flag set), it should use the git commit of the current branch as the docker image tag to pull from the docker registry.

Locally ran the Circle CI check-cross-platform script against all op components (op-node, op-proposer, etc) that now have the check-cross-platform job applied to them in the Circle CI config.yml. Confirmed that each one runs successfully when given a docker tag that exists in the docker registry.

Metadata

@bitwiseguy bitwiseguy marked this pull request as ready for review April 30, 2024 18:18
Copy link
Contributor

coderabbitai bot commented Apr 30, 2024

Walkthrough

Walkthrough

The modifications across various configuration files and Makefiles primarily focus on standardizing the versioning system (GIT_VERSION) and enhancing Docker build processes. The changes introduce dynamic version assignment, improve Docker image tagging, and streamline build dependencies. New build stages and targets are added to Dockerfiles to accommodate different components, enhancing modularity and compatibility checks across platforms.

Changes

Files Summary
.circleci/config.yml Updated to handle dynamic GIT_VERSION, reorganized workflows, and improved Docker image handling.
Makefile in various directories Standardized version declaration to allow overrides and ensure reproducibility.
docker-bake.hcl, .../docker-compose.yml, ops/docker/.../Dockerfile Updated Docker configurations and paths, introduced new targets, and set dynamic versioning for services.
cannon/Makefile, op-bootnode/Makefile, op-challenger/Makefile, op-conductor/Makefile, op-dispute-mon/Makefile, op-heartbeat/Makefile, op-plasma/Makefile, op-program/Makefile Changed version declarations to be dynamic, allowing potential overrides and ensuring consistent build environments.

Recent Review Details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits Files that changed from the base of the PR and between 10cf486 and 0cbd293.
Files selected for processing (1)
  • docker-bake.hcl (2 hunks)
Files skipped from review as they are similar to previous changes (1)
  • docker-bake.hcl

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Out of diff range and nitpick comments (5)
docker-bake.hcl (1)

[!TIP]
Codebase Verification

The verification process has confirmed that most of the new variables introduced in the docker-bake.hcl file are correctly used in the Docker build targets. However, there are exceptions:

  • IMAGE_TAGS is not used in the targets.
  • PLATFORMS is not used in the targets.

These findings suggest that while the majority of the variables are implemented correctly, there are issues with how IMAGE_TAGS and PLATFORMS are utilized, or potentially not utilized, within the build targets. This could lead to unexpected behavior or errors during the Docker image build process if these variables are intended to be used but are not correctly referenced.

Action Items:

  • Review the intended use of IMAGE_TAGS and PLATFORMS to ensure they are either removed if unnecessary or correctly implemented in the Docker build targets.
Analysis chain

Line range hint 17-212: Verify that all new variables introduced in docker-bake.hcl are correctly used in the Docker build targets and that their default values are appropriate.

Scripts Executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify the usage of new variables in Docker build targets.
# Test: Check each target in `docker-bake.hcl` for correct usage of variables.
echo "Check each Docker build target for correct variable usage."

Length of output: 126



Script:

#!/bin/bash
# Description: Verify the usage of new variables in Docker build targets by checking their presence in the target definitions.
# Extracting the variable names and checking their usage in the target definitions.

# List all variable names
variable_names=$(grep -oP 'variable "\K[^"]+' docker-bake.hcl)

# Check each variable usage in the target definitions
for var in $variable_names; do
    echo "Checking usage of variable: $var"
    grep -q "\${$var}" docker-bake.hcl && echo "$var is used in the targets." || echo "$var is NOT used in the targets."
done

Length of output: 3304

.circleci/config.yml (4)

1838-1838: Clarify the comment regarding the non-inclusion of op-program in the devnet.

Consider enhancing the comment to explain why op-program is not part of the devnet, as this could be useful for future maintainers.


1848-1848: Clarify the comment regarding the non-inclusion of op-challenger in the devnet.

Consider enhancing the comment to explain why op-challenger is not part of the devnet, as this could be useful for future maintainers.


1858-1858: Clarify the comment regarding the non-inclusion of op-conductor in the devnet.

Consider enhancing the comment to explain why op-conductor is not part of the devnet, as this could be useful for future maintainers.


1863-1863: Clarify the comment regarding the non-inclusion of op-heartbeat in the devnet.

Consider enhancing the comment to explain why op-heartbeat is not part of the devnet, as this could be useful for future maintainers.

Makefile Show resolved Hide resolved
@tynes
Copy link
Contributor

tynes commented May 1, 2024

🔴 However, the devnet-fault-proofs job took ~4m45s for the Bring up the stack step (mostly due to increased time for building op-challenger image) on this branch vs. ~1m45s for that same step that used the old docker build flow.

We have been thinking about removing the devnet from the monorepo and finding an owner for it as right now nobody truly owns the code

@protolambda
Copy link
Contributor

~4m45s for the Bring up the stack step

With docker-layer-caching and the use of the go-mod and go-build cache subsequent runs should be faster. On a fresh run it may take longer.

@bitwiseguy
Copy link
Contributor Author

@protolambda @tynes fixed the issue of slow build time during devnet tests with the following commits:

The build time was slow because I wasn't using the op-challenger.tar file created by the op-challenger-docker-build step. Bring up the stack step is now back down to 1m45 sec.

@bitwiseguy
Copy link
Contributor Author

@protolambda would it be worth me spending time to setup an automated Circle CI test to check that the docker images run on x86 machines? Or is that something you want to keep as a manual check?

@protolambda
Copy link
Contributor

@bitwiseguy nice work on getting the docker change back into shape!

Circle CI test to check that the docker images run on x86 machines?

You mean Arm machines? CI already runs in x86 I believe. Most of the dev machines run apple m1/m2, with a few exceptions. I think it makes sense to test ARM (circle CI should have an option to select the type of machine), but just a sanity check would be enough, like running with the --version flag, for each of the docker images. The hard part is that we want to build and publish the images on x86, since cross-building in one place avoids complicating deployment with needing to publish from different types of machines, but then pull and run on the arm machine.

Copy link
Contributor

semgrep-app bot commented May 3, 2024

Semgrep found 3 sol-style-return-arg-fmt findings:

  • packages/contracts-bedrock/scripts/L2Genesis.s.sol
  • packages/contracts-bedrock/scripts/Deployer.sol
  • packages/contracts-bedrock/scripts/Config.sol

Named return arguments to functions must be appended with an underscore (_)

Ignore this finding from sol-style-return-arg-fmt.

Copy link
Contributor

@protolambda protolambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@bitwiseguy bitwiseguy added this pull request to the merge queue May 6, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 6, 2024
@bitwiseguy bitwiseguy added this pull request to the merge queue May 6, 2024
Merged via the queue into develop with commit c67c67a May 6, 2024
71 of 72 checks passed
@bitwiseguy bitwiseguy deleted the ss/docker-targets-v2 branch May 6, 2024 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants