HARMONY-2244:Excessive alert emails from Kubernetes cluster#845
HARMONY-2244:Excessive alert emails from Kubernetes cluster#845blackone-sudo merged 2 commits intomainfrom
Conversation
blackone-sudo
requested review from
chris-durbin,
flamingbear,
indiejames and
ygliuvt
as code owners
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📝 WalkthroughWalkthroughUpdated alert-manager route Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
config/alert-manager.yaml (1)
56-56:⚠️ Potential issue | 🟡 MinorUpdate AlertManager image to a newer version.
The image
prom/alertmanager:v0.23.0is approximately 4–5 years old. The current stable release is v0.31.0 (released February 2, 2026), with multiple minor and patch versions released since v0.23.0. Consider updating to the latest version to gain bug fixes, security patches, and performance improvements.
🧹 Nitpick comments (1)
config/alert-manager.yaml (1)
16-16: Significant increase in repeat interval from 2h to 168h (7 days).This change effectively suppresses duplicate alerts for a week. While this addresses alert fatigue from pods stuck in Pending/Unknown states, consider the trade-off: if an issue persists unresolved, teams won't receive reminder notifications for 7 days.
A middle-ground like
24hor48hmight balance noise reduction while still providing periodic reminders for unresolved issues. However, if the current alert volume is severe and resolution workflows are well-established, 168h may be acceptable.
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In `@docs/guides/develop.md`:
- Around line 19-20: The phrase "command line interface" should be corrected to
use the compound modifier "command-line interface"; update the second bullet
that currently reads "Install
[kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command line
interface to Kubernetes." to "Install
[kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command-line
interface to Kubernetes." so the document follows the compound modifier grammar
rule.
- Line 89: Replace the asterisk-based emphasis around the leading "Tip" to use
underscores to satisfy MD049: locate the "*Tip*:" occurrence in the
docs/guides/develop.md content and change it to "_Tip_:" (leave surrounding
inline code spans like `gyp ERR!`, `v8.4.1`, and `pip install setuptools`
unchanged).
- Line 116: Update the phrase "Mac OS X specific env:" to use a hyphenated
compound modifier, i.e., change the text token "Mac OS X specific env" (found in
the docs/guides/develop.md line containing that phrase) to "Mac OS X-specific
env" so the compound modifier before the noun is correctly hyphenated.
- Line 35: Update the docs line that currently reads "Python version 3.13" to
reflect the supported range by changing it to "Python 3.10+" so the guide
accurately states support for Python 3.10–3.13+ (locate the text "Python version
3.13" in docs/guides/develop.md and replace it with "Python 3.10+").
| * Install [minikube](https://kubernetes.io/docs/tasks/tools/#minikube), a single-node Kubernetes cluster useful for local linux development. | ||
| * Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command line interface to Kubernetes. |
There was a problem hiding this comment.
Use “command-line” as a compound modifier.
This matches the grammar rule and the static lint hint.
✏️ Proposed fix
-* Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command line interface to Kubernetes.
+* Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command-line interface to Kubernetes.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| * Install [minikube](https://kubernetes.io/docs/tasks/tools/#minikube), a single-node Kubernetes cluster useful for local linux development. | |
| * Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command line interface to Kubernetes. | |
| * Install [minikube](https://kubernetes.io/docs/tasks/tools/#minikube), a single-node Kubernetes cluster useful for local linux development. | |
| * Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command-line interface to Kubernetes. |
🧰 Tools
🪛 LanguageTool
[grammar] ~20-~20: Use a hyphen to join words.
Context: ...io/docs/tasks/tools/#kubectl), a command line interface to Kubernetes. * [Docker ...
(QB_NEW_EN_HYPHEN)
🤖 Prompt for AI Agents
In `@docs/guides/develop.md` around lines 19 - 20, The phrase "command line
interface" should be corrected to use the compound modifier "command-line
interface"; update the second bullet that currently reads "Install
[kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command line
interface to Kubernetes." to "Install
[kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), a command-line
interface to Kubernetes." so the document follows the compound modifier grammar
rule.
|
|
||
| Optional: | ||
| * [Python](https://www.python.org) version 3.11 - Useful for locally running and testing harmony-docker and other backend services | ||
| * [Python](https://www.python.org) version 3.13 - Useful for locally running and testing harmony-docker and other backend services |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check workload/requirements.txt for Python version specifications
echo "=== Checking workload/requirements.txt ==="
if [ -f "workload/requirements.txt" ]; then
cat -n workload/requirements.txt | head -20
else
echo "File not found, searching for requirements files..."
find . -name "requirements.txt" -type f 2>/dev/null | head -10
fiRepository: nasa/harmony
Length of output: 353
🏁 Script executed:
#!/bin/bash
# Check docs/guides/develop.md at line 35
echo "=== Checking docs/guides/develop.md ==="
if [ -f "docs/guides/develop.md" ]; then
sed -n '30,40p' docs/guides/develop.md | cat -n
else
echo "File not found, searching for develop.md..."
find . -name "develop.md" -type f 2>/dev/null
fiRepository: nasa/harmony
Length of output: 677
🏁 Script executed:
#!/bin/bash
# Search for all Python version references in documentation
echo "=== Python version references in docs ==="
rg -n "Python.*3\.(10|11|12|13)" docs/ 2>/dev/null || echo "No matches in docs/"
rg -n "python.*3\.(10|11|12|13)" docs/ 2>/dev/null || echo "No matches in docs/"Repository: nasa/harmony
Length of output: 411
🏁 Script executed:
#!/bin/bash
# Search for Python version constraints in setup.py or pyproject.toml
echo "=== Checking for Python version constraints ==="
rg -n "python_requires|requires-python" -A2 2>/dev/null | head -20Repository: nasa/harmony
Length of output: 103
🏁 Script executed:
#!/bin/bash
# Look for setup.py or pyproject.toml files
fd -n "setup.py|pyproject.toml|setup.cfg" -type fRepository: nasa/harmony
Length of output: 286
🏁 Script executed:
#!/bin/bash
# Look for setup.py or pyproject.toml files (fix syntax)
fd "setup.py|pyproject.toml|setup.cfg" -type fRepository: nasa/harmony
Length of output: 226
🏁 Script executed:
#!/bin/bash
# Search for python_requires or requires-python in all files
rg "python_requires|requires-python" -A2Repository: nasa/harmony
Length of output: 38
🏁 Script executed:
#!/bin/bash
# Look for setup.py or pyproject.toml files
find . -name "setup.py" -o -name "pyproject.toml" -o -name "setup.cfg" 2>/dev/nullRepository: nasa/harmony
Length of output: 38
🏁 Script executed:
#!/bin/bash
# Check if there's a .python-version file
find . -name ".python-version" 2>/dev/nullRepository: nasa/harmony
Length of output: 38
🏁 Script executed:
#!/bin/bash
# Check GitHub workflow files or CI config for Python version specifications
find . -name "*.yml" -o -name "*.yaml" 2>/dev/null | head -20 | xargs grep -l "python" 2>/dev/nullRepository: nasa/harmony
Length of output: 82
🏁 Script executed:
#!/bin/bash
# Check the services-prod.yml file for Python version info
cat ./config/services-prod.yml | grep -A5 -B5 -i pythonRepository: nasa/harmony
Length of output: 498
🏁 Script executed:
#!/bin/bash
# Search more broadly in workflow and config files for Python versions
find . -path "./.git" -prune -o -type f \( -name "*.yml" -o -name "*.yaml" -o -name "Dockerfile*" \) -print 2>/dev/null | xargs grep -i "python.*3\.(10|11|12|13)" 2>/dev/null | head -20Repository: nasa/harmony
Length of output: 38
🏁 Script executed:
#!/bin/bash
# Check if there's a tox.ini or similar test config
find . -name "tox.ini" -o -name ".nvmrc" -o -name ".node-version" 2>/dev/nullRepository: nasa/harmony
Length of output: 64
🏁 Script executed:
#!/bin/bash
# Look for Dockerfile references with Python versions
find . -name "Dockerfile*" 2>/dev/null | xargs cat 2>/dev/null | grep -i "python" | head -20Repository: nasa/harmony
Length of output: 1074
🏁 Script executed:
#!/bin/bash
# Check for docker-compose files
find . -name "docker-compose*" 2>/dev/nullRepository: nasa/harmony
Length of output: 38
🏁 Script executed:
#!/bin/bash
# Let's check the main README for Python version guidance
if [ -f "README.md" ]; then
grep -n -i "python" README.md | head -10
fiRepository: nasa/harmony
Length of output: 38
Update Python version guidance to reflect full supported range.
The documentation currently specifies "Python version 3.13", but the project supports Python 3.10–3.13+. Change this to "Python 3.10+" to accurately reflect the supported versions.
✏️ Proposed fix
-* [Python](https://www.python.org) version 3.13 - Useful for locally running and testing harmony-docker and other backend services
+* [Python](https://www.python.org) version 3.10+ - Useful for locally running and testing harmony-docker and other backend services🤖 Prompt for AI Agents
In `@docs/guides/develop.md` at line 35, Update the docs line that currently reads
"Python version 3.13" to reflect the supported range by changing it to "Python
3.10+" so the guide accurately states support for Python 3.10–3.13+ (locate the
text "Python version 3.13" in docs/guides/develop.md and replace it with "Python
3.10+").
| ```bash | ||
| npm install | ||
| ``` | ||
| *Tip*: if you get `gyp ERR!` you might be working with a very old version (v8.4.1) to build sqlite and you will need to pip install `setuptools` into your python environment before continuing. |
There was a problem hiding this comment.
Fix emphasis style to satisfy markdownlint (MD049).
Switch to underscores for emphasis.
✏️ Proposed fix
-*Tip*: if you get `gyp ERR!` you might be working with a very old version (v8.4.1) to build sqlite and you will need to pip install `setuptools` into your python environment before continuing.
+_Tip_: if you get `gyp ERR!` you might be working with a very old version (v8.4.1) to build sqlite and you will need to pip install `setuptools` into your python environment before continuing.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| *Tip*: if you get `gyp ERR!` you might be working with a very old version (v8.4.1) to build sqlite and you will need to pip install `setuptools` into your python environment before continuing. | |
| _Tip_: if you get `gyp ERR!` you might be working with a very old version (v8.4.1) to build sqlite and you will need to pip install `setuptools` into your python environment before continuing. |
🧰 Tools
🪛 LanguageTool
[style] ~89-~89: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...et gyp ERR! you might be working with a very old version (v8.4.1) to build sqlite and yo...
(EN_WEAK_ADJECTIVE)
🪛 markdownlint-cli2 (0.20.0)
[warning] 89-89: Emphasis style
Expected: underscore; Actual: asterisk
(MD049, emphasis-style)
[warning] 89-89: Emphasis style
Expected: underscore; Actual: asterisk
(MD049, emphasis-style)
🤖 Prompt for AI Agents
In `@docs/guides/develop.md` at line 89, Replace the asterisk-based emphasis
around the leading "Tip" to use underscores to satisfy MD049: locate the
"*Tip*:" occurrence in the docs/guides/develop.md content and change it to
"_Tip_:" (leave surrounding inline code spans like `gyp ERR!`, `v8.4.1`, and
`pip install setuptools` unchanged).
| SERVICE_RUNNER_IMAGE=harmonyservices/service-runner:latest | ||
| ``` | ||
|
|
||
| Mac OS X specific env: |
There was a problem hiding this comment.
Hyphenate “Mac OS X-specific”.
Compound modifier before a noun should be hyphenated.
✏️ Proposed fix
-Mac OS X specific env:
+Mac OS X-specific env:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Mac OS X specific env: | |
| Mac OS X-specific env: |
🧰 Tools
🪛 LanguageTool
[grammar] ~116-~116: Use a hyphen to join words.
Context: ...ices/service-runner:latest Mac OS X specific env: text BACKEND_HOST=host...
(QB_NEW_EN_HYPHEN)
🤖 Prompt for AI Agents
In `@docs/guides/develop.md` at line 116, Update the phrase "Mac OS X specific
env:" to use a hyphenated compound modifier, i.e., change the text token "Mac OS
X specific env" (found in the docs/guides/develop.md line containing that
phrase) to "Mac OS X-specific env" so the compound modifier before the noun is
correctly hyphenated.
1b0dc05 to
068b6a4
Compare
2 participants
Jira Issue ID
HARMONY-2244
Description
Excessive alert emails from Kubernetes cluster (Pods in Pending/Unknown state)
Local Test Steps
PR Acceptance Checklist
Summary by CodeRabbit
Chores
Documentation