Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding datadog details to each lambda #36

Merged
merged 18 commits into from
Aug 14, 2023
Merged

Adding datadog details to each lambda #36

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
890df51
Adding datadog tracing and extention layer
devinmatte Jul 21, 2023
9ad2faa
Updating comment on layer cleanup
devinmatte Jul 21, 2023
44b7013
Pinning ddsource to shrink package, newer poetry
devinmatte Jul 23, 2023
828b6c2
Merge remote-tracking branch 'origin/main' into datadog-env-variables
devinmatte Aug 1, 2023
d86f775
Repinning datadog
devinmatte Aug 1, 2023
a20a714
Adding full tracing
devinmatte Aug 1, 2023
c1886e6
Closer to getting tracing working
devinmatte Aug 3, 2023
994bd5b
Don't check dd api key in code
devinmatte Aug 3, 2023
b7ef7fa
Merge remote-tracking branch 'origin/main' into datadog-env-variables
devinmatte Aug 4, 2023
2464102
Merge branch 'main' into datadog-env-variables
devinmatte Aug 10, 2023
9567596
Remove layer
devinmatte Aug 10, 2023
da4ff8f
Adding back tracing
devinmatte Aug 10, 2023
dedadd1
Run every day
devinmatte Aug 11, 2023
476dce5
Stagger lambda start times
devinmatte Aug 11, 2023
d53a409
Merge branch 'main' into datadog-env-variables
devinmatte Aug 11, 2023
353006e
Merge remote-tracking branch 'origin/main' into datadog-env-variables
devinmatte Aug 13, 2023
438f573
Adding dd env vars for new function
devinmatte Aug 14, 2023
3afde53
Merge remote-tracking branch 'refs/remotes/origin/datadog-env-variabl…
devinmatte Aug 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions deploy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
#!/bin/bash -x

if [[ -z "$MBTA_V2_API_KEY" ]]; then
echo "Must provide MBTA_V2_API_KEY in environment" 1>&2
if [[ -z "$MBTA_V2_API_KEY" || -z "$DD_API_KEY" ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I need a DD_API_KEY to deploy locally now? Are there any docs/instructions we need to update?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only for deployment. Probably need to update docs. You can use any DD_API_KEY from our account, anyone with the datadog account can get it easily in the UI

echo "Must provide MBTA_V2_API_KEY and DD_API_KEY in environment" 1>&2
exit 1
fi

STACK_NAME=ingestor
BUCKET=ingestor-lambda-deployments

# Identify the version and commit of the current deploy
GIT_VERSION=`git describe --tags --always`
GIT_SHA=`git rev-parse HEAD`
echo "Deploying version $GIT_VERSION | $GIT_SHA"

# Adding some datadog tags to get better data
DD_TAGS="git.commit.sha:$GIT_SHA,git.repository_url:github.com/transitmatters/data-ingestion"

poetry export -f requirements.txt --output ingestor/requirements.txt --without-hashes

pushd ingestor/
Expand All @@ -16,4 +24,4 @@ poetry run chalice package --stage prod --merge-template .chalice/resources.json
aws cloudformation package --template-file cfn/sam.json --s3-bucket $BUCKET --output-template-file cfn/packaged.yaml
aws cloudformation deploy --template-file cfn/packaged.yaml --stack-name $STACK_NAME \
--capabilities CAPABILITY_NAMED_IAM --no-fail-on-empty-changeset \
--parameter-overrides MbtaV2ApiKey=$MBTA_V2_API_KEY DDApiKey=$DD_API_KEY
--parameter-overrides MbtaV2ApiKey=$MBTA_V2_API_KEY DDApiKey=$DD_API_KEY GitVersion=$GIT_VERSION DDTags=$DD_TAGS
18 changes: 11 additions & 7 deletions ingestor/.chalice/config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
"autogen_policy": false,
"environment_variables": {
"DD_SITE": "datadoghq.com",
"DD_TRACE_ENABLED": "true",
"DD_ENV": "prod",
"DD_SERVICE": "ingestor"
"DD_SERVICE": "ingestor",
"DD_TRACE_ENABLED": "true"
},
"lambda_functions": {
"store_yesterday_alerts": {
Expand All @@ -20,14 +20,16 @@
"iam_policy_file": "policy-newtrains.json"
},
"bb_store_station_status": {
"iam_policy_file": "policy-bluebikes-store.json"
"iam_policy_file": "policy-bluebikes-store.json",
"lambda_timeout": 90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were these failing with default timeout?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep just slightly. Discovered it when trying to add tracing. They were hitting timeout every time

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there data missing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't look like it in s3, but hard to tell until we start using this data somewhere

},
"bb_store_station_info": {
"iam_policy_file": "policy-bluebikes-store.json"
"iam_policy_file": "policy-bluebikes-store.json",
"lambda_timeout": 90
},
"bb_calc_daily_stats": {
"iam_policy_file": "policy-bluebikes-calc.json",
"lambda_timeout": 60,
"lambda_timeout": 90,
"lambda_memory_size": 256
},
"store_landing_data": {
Expand All @@ -49,10 +51,12 @@
"lambda_timeout": 600
},
"update_delivered_trip_metrics": {
"iam_policy_file": "policy-delivered-trip-metrics-daily.json"
"iam_policy_file": "policy-delivered-trip-metrics-daily.json",
"lambda_timeout": 90
},
"update_delivered_trip_metrics_yesterday": {
"iam_policy_file": "policy-delivered-trip-metrics-daily.json"
"iam_policy_file": "policy-delivered-trip-metrics-daily.json",
"lambda_timeout": 90
},
"update_agg_trip_metrics": {
"iam_policy_file": "policy-agg-trip-metric-tables.json"
Expand Down
66 changes: 52 additions & 14 deletions ingestor/.chalice/resources.json
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to store env vars across the lambdas? Or we have to apply to each function?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best I could find was we need to do it per function. I couldn't find anything easier, I looked. Would love if we find something

Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@
"DDApiKey": {
"Type": "String",
"Description": "Datadog API key."
},
"DDTags": {
"Type": "String",
"Description": "Additional Datadog Tags"
},
"GitVersion": {
"Type": "String",
"Description": "Current Git Id"
}
},
"Resources": {
Expand All @@ -15,7 +23,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -25,7 +35,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -36,7 +48,9 @@
"Environment": {
"Variables": {
"MBTA_V2_API_KEY": { "Ref": "MbtaV2ApiKey" },
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -47,7 +61,9 @@
"Environment": {
"Variables": {
"MBTA_V2_API_KEY": { "Ref": "MbtaV2ApiKey" },
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -57,7 +73,10 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_LAMBDA_HANDLER": "app.bb_store_station_status",
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -67,7 +86,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -77,7 +98,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -87,7 +110,10 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_LAMBDA_HANDLER": "app.update_delivered_trip_metrics",
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -97,7 +123,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -107,7 +135,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -117,7 +147,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -127,7 +159,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -137,7 +171,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand All @@ -147,7 +183,9 @@
"Properties": {
"Environment": {
"Variables": {
"DD_API_KEY": { "Ref": "DDApiKey" }
"DD_API_KEY": { "Ref": "DDApiKey" },
"DD_VERSION": { "Ref": "GitVersion" },
"DD_TAGS": { "Ref": "DDTags" }
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions ingestor/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ def update_speed_restrictions(event):
speed_restrictions.update_speed_restrictions()


# 7:20am UTC -> 2:20/3:20am ET every Sunday
@app.schedule(Cron(20, 7, "?", "*", "SUN", "*"))
# 7:30am UTC -> 2:30/3:30am ET every day
@app.schedule(Cron(30, 7, "?", "*", "?", "*"))
def update_time_predictions(event):
predictions.update_predictions()

Expand Down
18 changes: 9 additions & 9 deletions poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ authors = ["TransitMatters Labs Team"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.10, <3.11"
python = "~3.10, <3.11"
requests = "^2.31.0"
pytz = "^2023.3"
boto3 = "^1.26.154"
boto3 = "^1.28.19"
numpy = "^1.24.2"
pandas = "^1.5.3"
geopy = "^2.2.0"
Expand All @@ -18,7 +18,7 @@ mbta-gtfs-sqlite = "^0.9.13"
boxsdk = "^3.7.1"
openpyxl = "^3.1.2"
dynamodb-json = "^1.3"
datadog_lambda = "^4.77.0"
datadog_lambda = "4.77.0"
ddtrace = "1.15.2"

[tool.poetry.dev-dependencies]
Expand All @@ -31,5 +31,5 @@ line-length = 120
target-version = ['py310']

[build-system]
requires = ["poetry-core>=1.0.0"]
requires = ["poetry-core>=1.5.0"]
build-backend = "poetry.core.masonry.api"