Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breakout npm_and_yarn, Move some dockerfiles #6486

Merged
merged 7 commits into from
Jan 20, 2023
Merged

Breakout npm_and_yarn, Move some dockerfiles #6486

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d4f75e4
add npm_and_yarn dockerfile
pavera Jan 19, 2023
0c7fdb2
a little work towards npm
pavera Jan 19, 2023
97c8ab6
Initial attempt at breaking out npm
pavera Jan 19, 2023
b6a13f9
remove accidentally commited file
pavera Jan 19, 2023
90170c1
adding more ecosystem images (#6485)
jakecoffman Jan 20, 2023
f9bb687
get helpers installed.
pavera Jan 20, 2023
4bd7dc1
bad order of ops here
pavera Jan 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/workflows/eco-branch-images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,15 @@ jobs:
fail-fast: false
matrix:
suite:
- { name: cargo, ecosystem: cargo }
- { name: docker, ecosystem: docker }
- { name: git_submodules, ecosystem: gitsubmodule }
- { name: github_actions, ecosystem: github-actions }
- { name: go_modules, ecosystem: gomod }
- { name: gradle, ecosystem: gradle }
- { name: maven, ecosystem: maven }
- { name: npm_and_yarn, ecosystem: npm }
- { name: nuget, ecosystem: nuget }
- { name: python, ecosystem: pip }
permissions:
contents: read
Expand Down Expand Up @@ -69,7 +76,7 @@ jobs:
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from ghcr.io/dependabot/dependabot-updater-core \
--cache-from ghcr.io/dependabot/dependabot-updater-${{ matrix.suite.ecosystem }} \
-f Dockerfile.${{ matrix.suite.name }} \
-f ${{ matrix.suite.name }}/Dockerfile \
.

- name: Push branch image
Expand Down
9 changes: 8 additions & 1 deletion .github/workflows/eco-latest-releases.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,15 @@ jobs:
fail-fast: false
matrix:
suite:
- { name: cargo, ecosystem: cargo }
- { name: docker, ecosystem: docker }
- { name: git_submodules, ecosystem: gitsubmodule }
- { name: github_actions, ecosystem: github-actions }
- { name: go_modules, ecosystem: gomod }
- { name: gradle, ecosystem: gradle }
- { name: maven, ecosystem: maven }
- { name: npm_and_yarn, ecosystem: npm }
- { name: nuget, ecosystem: nuget }
- { name: python, ecosystem: pip }
env:
TAG: ${{ github.sha }}
Expand Down Expand Up @@ -65,7 +72,7 @@ jobs:
-t "${UPDATER_IMAGE}${ECOSYSTEM}:$TAG" \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from "${UPDATER_IMAGE_MIRROR}${ECOSYSTEM}" \
-f Dockerfile.${NAME} \
-f ${NAME}/Dockerfile \
.

- name: Log in to GHCR
Expand Down
62 changes: 55 additions & 7 deletions .github/workflows/eco-smoke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,17 @@ jobs:
fail-fast: false
matrix:
suite:
- { path: cargo, name: cargo, ecosystem: cargo }
- { path: docker, name: docker, ecosystem: docker }
- { path: git_submodules, name: submodules, ecosystem: gitsubmodule }
- { path: github_actions, name: actions, ecosystem: github-actions }
- { path: go_modules, name: go, ecosystem: gomod }
- { path: gradle, name: gradle, ecosystem: gradle }
- { path: maven, name: maven, ecosystem: maven }
- { path: npm_and_yarn, name: npm, ecosystem: npm}
- { path: npm_and_yarn, name: npm-remove-transitive, ecosystem: npm}
- { path: npm_and_yarn, name: yarn-berry, ecosystem: npm}
- { path: nuget, name: nuget, ecosystem: nuget }
- { path: python, name: pip, ecosystem: pip }
- { path: python, name: pipenv, ecosystem: pip}
- { path: python, name: pip-compile, ecosystem: pip }
Expand All @@ -31,42 +40,81 @@ jobs:
id: changes
with:
filters: |
cargo:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'cargo/**'
docker:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'docker/**'
git_submodules:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'git_submodules/**'
github_actions:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'github_actions/**'
go:
- Dockerfile.go_modules
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'go_modules/**'
gradle:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'gradle/**'
maven:
- Dockerfile.maven
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'maven/**'
npm:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'npm_and_yarn/**'
'npm-remove-transitive':
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'npm_and_yarn/**'
nuget:
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'nuget/**'
pip:
- Dockerfile.python
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'python/**'
'pip-compile':
- Dockerfile.python
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'python/**'
pipenv:
- Dockerfile.python
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'python/**'
poetry:
- Dockerfile.python
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'python/**'
'yarn-berry':
- Dockerfile.updater-core
- 'common/**'
- 'updater/**'
- 'npm_and_yarn/**'

- name: Download CLI and test
if: steps.changes.outputs[matrix.suite.name] == 'true'
Expand Down Expand Up @@ -108,7 +156,7 @@ jobs:
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from ghcr.io/dependabot/dependabot-updater-core \
--cache-from ghcr.io/dependabot/dependabot-updater-gomod \
-f Dockerfile.${{ matrix.suite.path }} \
-f ${{ matrix.suite.path }}/Dockerfile \
.

- name: ${{ matrix.suite.name }}
Expand Down
57 changes: 53 additions & 4 deletions .github/workflows/ecosystem-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,18 @@ jobs:
fail-fast: false
matrix:
suite:
- { path: cargo, name: cargo, ecosystem: cargo }
- { path: docker, name: docker, ecosystem: docker }
- { path: git_submodules, name: git_submodules, ecosystem: submodule }
- { path: github_actions, name: github_actions, ecosystem: github-actions }
- { path: go_modules, name: go_module, ci_node_total: 2, ci_node_index: 0, ecosystem: gomod }
- { path: go_modules, name: go_module, ci_node_total: 2, ci_node_index: 1, ecosystem: gomod }
- { path: gradle, name: gradle, ecosystem: gradle }
- { path: maven, name: maven, ecosystem: maven }
- { path: npm_and_yarn, name: npm_and_yarn, ci_node_total: 3, ci_node_index: 0, ecosystem: npm }
- { path: npm_and_yarn, name: npm_and_yarn, ci_node_total: 3, ci_node_index: 1, ecosystem: npm }
- { path: npm_and_yarn, name: npm_and_yarn, ci_node_total: 3, ci_node_index: 2, ecosystem: npm }
- { path: nuget, name: nuget, ecosystem: nuget }
- { path: python, name: python, ci_node_total: 2, ci_node_index: 0, ecosystem: pip }
- { path: python, name: python, ci_node_total: 2, ci_node_index: 1, ecosystem: pip}
- { path: python, name: python_slow, ci_node_total: 2, ci_node_index: 0, ecosystem: pip }
Expand All @@ -30,21 +39,61 @@ jobs:
id: changes
with:
filters: |
cargo:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'cargo/**'
docker:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'docker/**'
git_submodules:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'git_submodules/**'
github_actions:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'github_actions/**'
gradle:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'maven/**'
- 'gradle/**'
nuget:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'nuget/**'
python:
- Dockerfile.updater-core
- Dockerfile.python
- 'common/**'
- 'python/**'
- '.github/workflows/ecosystem-ci.yml'
maven:
- Dockerfile.updater-core
- Dockerfile.maven
- 'common/**'
- 'maven/**'
- '.github/workflows/ecosystem-ci.yml'
npm_and_yarn:
- Dockerfile.updater-core
- 'common/**'
- 'omnibus/**'
- '.github/workflows/ecosystem-ci.yml'
- 'npm_and_yarn/**'
go_modules:
- Dockerfile.updater-core
- Dockerfile.go_modules
- 'common/**'
- 'go_modules/**'
- '.github/workflows/ecosystem-ci.yml'
Expand All @@ -70,7 +119,7 @@ jobs:
-t "dependabot/dependabot-updater-${{ matrix.suite.ecosystem }}:latest" \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from ghcr.io/dependabot/dependabot-updater-${{ matrix.suite.ecosystem }} \
-f Dockerfile.${{ matrix.suite.path }} \
-f ${{ matrix.suite.path }}/Dockerfile \
.

- name: Run ${{ matrix.suite.name }} tests
Expand Down
2 changes: 2 additions & 0 deletions Dockerfile.updater-core
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ bundle install

FROM ubuntu:20.04

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

ENV DEBIAN_FRONTEND="noninteractive" \
LC_ALL="en_US.UTF-8" \
LANG="en_US.UTF-8"
Expand Down
20 changes: 20 additions & 0 deletions cargo/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM ghcr.io/dependabot/dependabot-updater-core

USER root
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends curl

# Install Rust
ENV RUSTUP_HOME=/opt/rust \
CARGO_HOME=/opt/rust \
PATH="${PATH}:/opt/rust/bin"
RUN mkdir -p "$RUSTUP_HOME" && chown dependabot:dependabot "$RUSTUP_HOME"

USER dependabot
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain 1.66.0 --profile minimal

# Configure cargo to use Git CLI so the Git shim works
RUN mkdir -p ~/.cargo && printf "[net]\ngit-fetch-with-cli = true\n" >> ~/.cargo/config.toml

COPY --chown=dependabot:dependabot cargo /home/dependabot/cargo
5 changes: 5 additions & 0 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
FROM ghcr.io/dependabot/dependabot-updater-core

USER dependabot

COPY --chown=dependabot:dependabot docker /home/dependabot/docker
5 changes: 5 additions & 0 deletions git_submodules/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
FROM ghcr.io/dependabot/dependabot-updater-core

USER dependabot

COPY --chown=dependabot:dependabot git_submodules /home/dependabot/git_submodules
5 changes: 5 additions & 0 deletions github_actions/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
FROM ghcr.io/dependabot/dependabot-updater-core

USER dependabot

COPY --chown=dependabot:dependabot github_actions /home/dependabot/github_actions
0 Dockerfile.go_modules → go_modules/Dockerfile
View file
Open in desktop
File renamed without changes.
6 changes: 6 additions & 0 deletions gradle/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
FROM ghcr.io/dependabot/dependabot-updater-core

USER dependabot

COPY --chown=dependabot:dependabot maven /home/dependabot/maven
COPY --chown=dependabot:dependabot gradle /home/dependabot/gradle
0 Dockerfile.maven → maven/Dockerfile
View file
Open in desktop
File renamed without changes.
45 changes: 45 additions & 0 deletions npm_and_yarn/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
FROM ghcr.io/dependabot/dependabot-updater-core
ENV DEPENDABOT_HOME /home/dependabot
### JAVASCRIPT

# Install Node and npm
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
curl \
&& curl -sL https://deb.nodesource.com/setup_16.x | bash - \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
nodejs \
&& rm -rf /var/lib/apt/lists/* \
&& npm install -g npm@8.19.2 \
&& rm -rf ~/.npm

# Install yarn berry and set it to a stable version
RUN corepack enable \
&& corepack prepare yarn@3.2.3 --activate

USER dependabot
ENV DEPENDABOT_NATIVE_HELPERS_PATH="/opt"
COPY --chown=dependabot:dependabot npm_and_yarn/helpers /opt/npm_and_yarn/helpers
RUN bash /opt/npm_and_yarn/helpers/build

# START: HACKY WORKAROUND FOR NPM GIT INSTALLS SPAWNING CHILD PROCESS

# TODO: Remove these hacks once we've deprecated npm 6 support as it no longer
# spawns a child process to npm install git dependencies.

# Create the config file manually intead of using yarn/npm config set as this
# executes the package manager outputs to every job log
COPY --chown=dependabot:dependabot updater/config/.yarnrc updater/config/.npmrc $DEPENDABOT_HOME/

# For Yarn Berry we can set this via an environment variable
ENV NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt

# END: HACKY WORKAROUND FOR NPM GIT INSTALLS SPAWNING CHILD PROCESS

# Our native helpers pull in yarn 1, so we need to reset the version globally to
# 3.2.3.
RUN corepack prepare yarn@3.2.3 --activate

COPY --chown=dependabot:dependabot npm_and_yarn ${DEPENDABOT_HOME}/npm_and_yarn
5 changes: 5 additions & 0 deletions nuget/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
FROM ghcr.io/dependabot/dependabot-updater-core

USER dependabot

COPY --chown=dependabot:dependabot nuget /home/dependabot/nuget
0 Dockerfile.python → python/Dockerfile
View file
Open in desktop
File renamed without changes.