Skip to content

Commit

Permalink
Upgrade CDK (v2.119), SAM (v1.107), Jinja2 (v3.1.3), and others to la…
Browse files Browse the repository at this point in the history
…test compatible version (#676)

* Upgrade CDK (v2.113), SAM (v1.103), and others to latest compatible version

**Why?**

To use the latest versions as made available in the meanwhile.

**What?**

* Upgrade CDK (v2.113), SAM (v1.103), and others to latest compatible version
* Remove Python based awscli v1, rely on default AWS CLI part of CodeBuild
  instead.
* Fix Terraform Enabled Parameter Store test
* Hide logs emitted by tests.

* Upgrade CDK (v2.115), SAM (v1.105), Python (v3.12) and others to latest compatible

**Why?**

To use the latest versions as made available in the meanwhile.

**What?**

* Upgrade CDK (v2.115), SAM (v1.105), and others to latest compatible version
* Update to Python v3.12.
* Ensure Regular Expressions use raw strings where required.

* Upgrade CDK (v2.118.0) and ARM64

**Why?**

To use the latest versions as made available in the meanwhile.

**What?**

* Upgrade CDK (v2.118.0).
* Upgrade AWS SAM CLI to v1.107.0.
* Switch to the ARM64 architecture for Lambda functions.
* Add the new required ImageId property for Cloud9 instances in the service
  catalog sample.

* Undo ARM64 for all Lambda functions

Reverting, due to `ruamel-yaml-clib==0.2.8(wheel)` dependency error.

* Bump jinja2

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.2...3.1.3)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump jinja2 from 3.1.2 to 3.1.3 in /src/lambda_codebase/initial_commit

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.2...3.1.3)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Upgrade ADF Workflow to Python 3.12

* Fix repeated headers blocker in providers guide

* Upgrade to CDK v2.119

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
  • Loading branch information
sbkok and dependabot[bot] authored Jan 12, 2024
1 parent 0c038b2 commit 8d0445b
Show file tree
Hide file tree
Showing 55 changed files with 235 additions and 143 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/adf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9"]
python-version: ["3.12"]

steps:
- name: Checkout Repo
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ test:

lint:
# Linter performs static analysis to catch latent bugs
find src -iname "*.py" -not -path "src/.aws-sam/*" | xargs pylint --rcfile .pylintrc
find src -iname "*.py" -not -path "src/.aws-sam/*" | xargs pylint --verbose --rcfile .pylintrc
find src -iname "*.yml" -o -iname "*.yaml" -not -path "src/.aws-sam/*" | xargs yamllint -c .yamllint.yml
cfn-lint

Expand Down
3 changes: 3 additions & 0 deletions docs/providers-guide.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Providers Guide

<!-- markdownlint-disable MD024 -->
<!-- ^ Allow repeated headers to be used in this file -->

Provider types and their properties can be defined as default config for a
pipeline. But also at the stage level of a pipeline to structure the source,
build, test, approval, deploy or invoke actions.
Expand Down
4 changes: 2 additions & 2 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -954,8 +954,8 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
nodejs: 18
python: 3.12
nodejs: 20
pre_build:
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
Expand Down
14 changes: 7 additions & 7 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
cfn-lint==0.78.2
isort==5.12.0
cfn-lint==0.83.8
isort==5.13.2
mock==5.1.0
pylint==2.17.4
pytest~=7.4.0
pytest-cov==3.0.0
tox==3.28.0
yamllint==1.32.0
pylint==3.0.3
pytest~=7.4.4
pytest-cov==4.1.0
tox==4.11.4
yamllint==1.33.0
7 changes: 4 additions & 3 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
astroid==2.15.6
boto3==1.28.8
botocore==1.31.8
astroid==3.0.2
aws-sam-cli==1.107.0
boto3==1.34.17
botocore==1.34.17
pyyaml~=6.0.1
schema==0.7.5
4 changes: 2 additions & 2 deletions samples/sample-cdk-app/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
nodejs: 18
python: 3.12
nodejs: 20
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-codebuild-vpc/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
# It will connect through the VPC to fetch all the resources.
# Make sure the subnets and security groups are configured such that
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-codebuild-vpc/testspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12

build:
commands:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-ec2-with-codedeploy/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-ecr-repository/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-ecs-cluster/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-expunge-vpc/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-expunge-vpc/template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Resources:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt LambdaVPCPolicyRole.Arn
Runtime: python3.10
Runtime: python3.12
Timeout: 600
Environment:
Variables:
Expand Down
3 changes: 1 addition & 2 deletions samples/sample-fargate-node-app/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
docker: 20
python: 3.11
python: 3.12

build:
commands:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-iam/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-mono-repo/apps/alpha/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ env:
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- cd $INFRASTRUCTURE_ROOT_DIR
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-mono-repo/apps/beta/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ env:
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- cd $INFRASTRUCTURE_ROOT_DIR
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
Expand Down
4 changes: 2 additions & 2 deletions samples/sample-rdk-rules/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
nodejs: 18
python: 3.12
nodejs: 20
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"Parameters": {
"RuleName": "EC2_CHECKS_TERMINATION_PROTECTION_ADF",
"Description": "EC2_CHECKS_TERMINATION_PROTECTION_ADF",
"SourceRuntime": "python3.10",
"SourceRuntime": "python3.12",
"CodeKey": "EC2_CHECKS_TERMINATION_PROTECTION_ADFeu-central-1.zip",
"InputParameters": "{}",
"OptionalParameters": "{}",
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-rdk-rules/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
s3==3.0.0
boto3==1.28.8
boto3==1.34.17
argparse==1.4.0
2 changes: 1 addition & 1 deletion samples/sample-serverless-app/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12

build:
commands:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-serverless-app/template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Resources:
Type: 'AWS::Serverless::Function'
Properties:
Handler: lambda_function.handler
Runtime: python3.10
Runtime: python3.12
CodeUri: .
Description: Sample Lambda Function
MemorySize: 128
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-service-catalog-product/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Resources:
AutomaticStopTimeMinutes: !Ref AutomaticStopTimeInMinutes
Description: !Ref InstanceDescription
InstanceType: !Ref InstanceType
ImageId: 'amazonlinux-2023-x86_64'
Name: !Ref InstanceName
OwnerArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:user/${UserName}" # In this sample case 'sample-developer' from the IAM stack can be used here
SubnetId:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-terraform/tf_apply.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ env:
phases:
install:
runtime-versions:
python: 3.11
python: 3.12

build:
commands:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-terraform/tf_destroy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ env:
phases:
install:
runtime-versions:
python: 3.9
python: 3.12

build:
commands:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-terraform/tf_plan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ env:
phases:
install:
runtime-versions:
python: 3.11
python: 3.12

build:
commands:
Expand Down
2 changes: 1 addition & 1 deletion samples/sample-vpc/buildspec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
phases:
install:
runtime-versions:
python: 3.11
python: 3.12
commands:
- aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q
Expand Down
4 changes: 3 additions & 1 deletion src/lambda_codebase/account/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
deployment account if required.
"""

import os
from typing import Mapping, Any, Tuple
from dataclasses import dataclass, asdict
import logging
Expand All @@ -28,7 +29,8 @@
ORGANIZATION_CLIENT = boto3.client("organizations")
SSM_CLIENT = boto3.client("ssm")
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
LOGGER.setLevel(os.environ.get("ADF_LOG_LEVEL", logging.INFO))
logging.basicConfig(level=logging.INFO)
MAX_RETRIES = 120 # => 120 retries * 5 seconds = 10 minutes


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ def process_account_list(all_accounts, accounts_in_file):


def sanitize_account_name_for_snf(account_name):
return re.sub("[^a-zA-Z0-9_]", "_", account_name[:30])
return re.sub(r"[^a-zA-Z0-9_]", "_", account_name[:30])


def start_executions(
Expand Down
3 changes: 1 addition & 2 deletions src/lambda_codebase/account_processing/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
aws-xray-sdk==2.12.0
aws-xray-sdk==2.12.1
pyyaml~=6.0.1
wrapt==1.14.1 # https://github.com/aws/aws-lambda-builders/issues/302
7 changes: 5 additions & 2 deletions src/lambda_codebase/cross_region_bucket/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@
"""


import os
from typing import Mapping, Any, Tuple, MutableMapping
from dataclasses import dataclass, asdict
import logging
import json
import secrets
import string # pylint: disable=deprecated-module # https://www.logilab.org/ticket/2481
import string # pylint: disable=deprecated-module
# ^ https://www.logilab.org/ticket/2481
import boto3
from cfn_custom_resource import ( # pylint: disable=unused-import
lambda_handler,
Expand All @@ -34,7 +36,8 @@

# Globals:
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
LOGGER.setLevel(os.environ.get("ADF_LOG_LEVEL", logging.INFO))
logging.basicConfig(level=logging.INFO)
S3CLIENTS: MutableMapping[Region, S3Client] = {}
SSM_CLIENT = boto3.client("ssm")

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,7 @@ Parameters:
ComputeType:
Description: The Compute Type to use for AWS CodeBuild
Type: String
# BUILD_GENERAL1_LARGE - For threading with large amounts of pipelines
# this is the most effective default:
Default: "BUILD_GENERAL1_LARGE"
Default: "BUILD_GENERAL1_SMALL"
AllowedValues:
- "BUILD_GENERAL1_SMALL" # 3 GB memory, 2 vCPU
- "BUILD_GENERAL1_MEDIUM" # 7 GB memory, 4 vCPU
Expand Down Expand Up @@ -75,15 +73,15 @@ Conditions:
Globals:
Function:
CodeUri: lambda_codebase
Runtime: python3.10
Runtime: python3.12

Resources:
LambdaLayerVersion:
Type: "AWS::Serverless::LayerVersion"
Properties:
ContentUri: "../../adf-build/shared/"
CompatibleRuntimes:
- python3.10
- python3.12
Description: "Shared Lambda Layer between master and deployment account"
LayerName: shared_layer

Expand Down Expand Up @@ -689,8 +687,8 @@ Resources:
phases:
install:
runtime-versions:
python: 3.11
nodejs: 18
python: 3.12
nodejs: 20
commands:
- aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -r adf-build/helpers/requirements.txt -q -t ./adf-build
Expand Down
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
boto3==1.28.8
boto3==1.34.17
cfn-custom-resource~=1.0.1
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Jinja2==3.1.2
boto3==1.28.8
Jinja2==3.1.3
boto3==1.34.17
cfn-custom-resource~=1.0.1
markupsafe==2.1.3
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
pyyaml~=6.0.1
schema==0.7.5
tenacity==8.2.2
wrapt==1.14.1 # https://github.com/aws/aws-lambda-builders/issues/302
tenacity==8.2.3
Loading

0 comments on commit 8d0445b

Please sign in to comment.