Lab 12 submission - Advanced Kubernetes Resilience by N1qro · Pull Request #361 · inno-devops-labs/SRE-Intro

N1qro · 2026-07-17T20:57:25Z

Lab 12 - Advanced Kubernetes Resilience

Task 1 done - Scaled the application services, added four PodDisruptionBudgets, configured gateway topology spreading, and verified failover and eviction blocking.
Task 2 done - Added graceful gateway shutdown behavior and applied a concurrent PostgreSQL index migration under live load.
Bonus Task done - Executed the complete expand-and-contract rename from event_date to scheduled_at with three migrations and two events-service deploys.
Optional HPA observation not completed.

Summary

Scaled events, payments, and notifications to two replicas
Added PDBs for gateway, events, payments, and notifications
Added gateway topology spreading, preStop, readiness, and termination-grace settings
Verified zero gateway 5xx during pod failure and gateway restart tests
Added CREATE INDEX CONCURRENTLY migration with Alembic autocommit
Executed the five-step expand-and-contract migration under mixed load
Preserved the existing events API response while moving the database to scheduled_at
Verified the final schema, migration rollback path, seed compatibility, and zero 5xx delta
Documented commands, outputs, observations, and design answers in submissions/lab12.md

…eful degradation and resource analysis

Lab 1 submission - deploy, break, understand

Lab 2 submission - inspect, optimize, trace

Lab 3 submission - monitor, observe, define SLOs

Lab 4 submission - deploy, probe, helm

Lab 5 submission - CI/CD, GitOps, rollback

Lab 6 submission - alerting, responding

Fix CI manifest update push race

Lab 7 submission - Progressive Delivery

Lab 8 submission - Chaos Engineering

Lab 9 submission - Stateful Services and DB Reliability

Lab 10 submission - SRE Portfolio and Reliability Review

Lab 11 submission - Advanced Microservice Patterns

Copilot

Pull request overview

This PR delivers the Lab 12 “Advanced Kubernetes Resilience” submission by hardening the QuickTicket Kubernetes deployment (replicas, PDBs, rollout behavior), executing an online schema change workflow (expand/contract rename to scheduled_at), and documenting/validating the work with updated monitoring assets and lab writeups.

Changes:

Increased service resilience via replica scaling, PodDisruptionBudgets, and gateway Rollout termination/readiness/topology-spread adjustments.
Added Alembic migrations (including concurrent index work) and performed an expand-and-contract rename from event_date → scheduled_at, including seed data updates and events-service query updates.
Updated observability artifacts (Grafana dashboard panels, Prometheus rule wiring) and added supporting runbook/handbook + CI workflow updates.

Reviewed changes

Copilot reviewed 57 out of 58 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
submissions/runbooks/quickticket-handbook.md	New operational handbook/runbook content for QuickTicket.
submissions/lab1.md	Lab 1 submission writeup.
submissions/lab2.md	Lab 2 submission writeup.
submissions/lab3.md	Lab 3 submission writeup.
submissions/lab4.md	Lab 4 submission writeup.
submissions/lab5.md	Lab 5 submission writeup.
submissions/lab6.md	Lab 6 submission writeup.
submissions/lab7.md	Lab 7 submission writeup.
submissions/lab8.md	Lab 8 submission writeup.
submissions/lab9.md	Lab 9 submission writeup (migrations/backup/restore work).
submissions/lab10.md	Lab 10 submission writeup (reliability review/load testing).
submissions/lab11.md	Lab 11 submission writeup (notifications + resilience patterns).
submissions/lab12.md	Lab 12 submission writeup (advanced K8s resilience + migrations).
monitoring/prometheus/rules.yml	Adds/defines SLO-related recording rules.
monitoring/prometheus/prometheus.yml	Prometheus scrape config (loads rules file).
monitoring/grafana/dashboards/golden-signals.json	Replaces placeholder panels with latency/saturation/SLO panels.
migrations/versions/0001_baseline_pre_existing_schema.py	Alembic baseline revision.
migrations/versions/0002_add_email_column_to_events.py	Alembic migration: add `email` column.
migrations/versions/0003_index_events_event_date_concurrently.py	Alembic migration: concurrent index creation/drop for `event_date`.
migrations/versions/0004_add_events_scheduled_at.py	Alembic migration: add nullable `scheduled_at`.
migrations/versions/0005_backfill_events_scheduled_at.py	Alembic migration: backfill + enforce NOT NULL on `scheduled_at`.
migrations/versions/0006_drop_events_event_date.py	Alembic migration: drop `event_date` (contract step).
migrations/env.py	Alembic environment configuration.
migrations/README	Alembic migrations README.
migrations/script.py.mako	Alembic script template.
locustfile.py	In-cluster Locust scenario for gateway load testing.
k8s/redis.yaml	Redis Deployment/Service manifest.
k8s/postgres.yaml	Postgres Deployment/Service with PVC-backed storage.
k8s/events.yaml	Events Deployment/Service manifest (2 replicas, env/probes).
k8s/payments.yaml	Payments Deployment/Service manifest (scaled to 2 replicas).
k8s/notifications.yaml	Notifications Deployment/Service manifest (scaled to 2 replicas).
k8s/gateway.yaml	Gateway Rollout updates (graceful shutdown, topology spread, analysis steps).
k8s/pdb.yaml	Adds PDBs for gateway/events/payments/notifications.
k8s/analysis-template.yaml	Defines canary error-rate AnalysisTemplate.
k8s/backup-cronjob.yaml	Adds Postgres backup CronJob manifest.
k8s/chart/Chart.yaml	Helm chart metadata.
k8s/chart/values.yaml	Helm chart values for QuickTicket components.
k8s/chart/templates/gateway.yaml	Helm template for gateway resources.
k8s/chart/templates/events.yaml	Helm template for events resources.
k8s/chart/templates/payments.yaml	Helm template for payments resources.
k8s/chart/templates/postgres.yaml	Helm template for postgres resources.
k8s/chart/templates/redis.yaml	Helm template for redis resources.
docker-compose.monitoring.yaml	Wires Prometheus rules file into docker-compose monitoring setup.
app/seed.sql	Updates seed schema/data to use `scheduled_at` instead of `event_date`.
app/gateway/main.py	Implements retry/circuit-breaker/rate-limiter logic and adds payments failure handling.
app/events/main.py	Switches reads/order to `scheduled_at` (aliased to preserve API response shape).
app/notifications/main.py	Adds notifications service implementation and metrics.
app/notifications/requirements.txt	Notifications service dependencies.
app/notifications/Dockerfile	Builds notifications service image (non-root user).
app/gateway/Dockerfile	Updates gateway Dockerfile (non-root user).
app/events/Dockerfile	Updates events Dockerfile (non-root user).
app/payments/Dockerfile	Updates payments Dockerfile (non-root user).
app/gateway/.dockerignore	Adds dockerignore for gateway build context.
app/events/.dockerignore	Adds dockerignore for events build context.
app/payments/.dockerignore	Adds dockerignore for payments build context.
alembic.ini	Adds Alembic configuration.
.gitignore	Adjusts ignored paths to allow committing lab artifacts/manifests.
.github/workflows/ci.yml	Adds CI workflow to build/push images and update manifests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    except httpx.RequestError:
+        return JSONResponse(
+            status_code=503,
+            content={
+                "error": "payments_unavailable",
+                "message": "Payment service is temporarily down. Your reservation is held — try again in a few minutes.",
+                "reservation_id": reservation_id,
+            },
+        )
    except httpx.TimeoutException:
        raise HTTPException(504, "Payment service timeout")


+EXPOSE 8080
+RUN addgroup --system app && adduser --system --ingroup app app
+USER app
+
 EXPOSE 8080


+EXPOSE 8081
+RUN addgroup --system app && adduser --system --ingroup app app
+USER app
+
 EXPOSE 8081


+EXPOSE 8082
+RUN addgroup --system app && adduser --system --ingroup app app
+USER app
+
 EXPOSE 8082


+          image: quickticket-notifications:v1
+          imagePullPolicy: Never


+def downgrade() -> None:
+    op.add_column(
+        "events",
+        sa.Column("event_date", sa.TIMESTAMP(timezone=True), nullable=True),
+    )
+    op.execute("UPDATE events SET event_date = scheduled_at")
+    op.alter_column("events", "event_date", nullable=False)


N1qro and others added 30 commits June 12, 2026 17:34

docs(lab1): add submission — deploy and failure exploration with grac…

07cf15d

…eful degradation and resource analysis

docs(lab2): add lab2 — Docker inspection and optimization

e559aad

Merge pull request #1 from N1qro/feature/lab1

939403a

Lab 1 submission - deploy, break, understand

Merge pull request #2 from N1qro/feature/lab2

2d6926a

Lab 2 submission - inspect, optimize, trace

feat(lab3): add monitoring config and SLO definitions

1649b7f

docs(lab4): add lab4 — K8s manifests and deployment

40a01d3

Merge pull request #3 from N1qro/feature/lab3

82d7fbc

Lab 3 submission - monitor, observe, define SLOs

Merge pull request #4 from N1qro/feature/lab4

0d33d52

Lab 4 submission - deploy, probe, helm

feat(lab5): add CI/CD GitOps pipeline

466bb4b

fix: gitignore was interfering with ci

cc09beb

ci: update image tags to cc09beb

1da7ad2

feat: deploy broken gateway image [skip ci]

7fb3804

revert: rollback broken gateway image [skip ci]

af94e3b

docs(lab5): complete CI/CD and GitOps report

5d126ef

ci: update image tags to 5d126ef

de51213

feat(lab6): add alerting config, runbook, and postmortem

e44b340

Merge pull request #5 from N1qro/feature/lab5

af08af1

Lab 5 submission - CI/CD, GitOps, rollback

Merge pull request #6 from N1qro/feature/lab6

6440472

Lab 6 submission - alerting, responding

ci: update image tags to 6440472

69e2df5

fix(ci): skip stale manifest updates

0115b03

feat(lab7): add gateway canary rollout

7e8e6bb

Merge pull request #8 from N1qro/fix/ci-push-race

2544c7d

Fix CI manifest update push race

ci: update image tags to 2544c7d

0086c8b

Merge pull request #7 from N1qro/feature/lab7

f5eeaf1

Lab 7 submission - Progressive Delivery

ci: update image tags to f5eeaf1

16d36f5

feat(lab8): add chaos engineering report

e8bab4d

Merge pull request #9 from N1qro/feature/lab8

bd0118c

Lab 8 submission - Chaos Engineering

ci: update image tags to bd0118c

e3768db

feat(lab9): add migrations and database reliability report

a0e6313

Merge pull request #10 from N1qro/feature/lab9

948b532

Lab 9 submission - Stateful Services and DB Reliability

github-actions and others added 8 commits June 29, 2026 17:43

ci: update image tags to 948b532

aa3fe94

feat(lab10): add reliability review and handbook

930801f

Merge pull request #11 from N1qro/feature/lab10

3387d68

Lab 10 submission - SRE Portfolio and Reliability Review

ci: update image tags to 3387d68

ce96296

feat(lab11): add resilience patterns and notifications service

025e154

Merge pull request #12 from N1qro/feature/lab11

f76fcf3

Lab 11 submission - Advanced Microservice Patterns

ci: update image tags to f76fcf3

9028d8e

feat(lab12): add kubernetes resilience and zero-downtime migrations

4a90d5a

Copilot AI review requested due to automatic review settings July 17, 2026 20:57

Copilot started reviewing on behalf of N1qro July 17, 2026 20:57 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Lab 12 submission - Advanced Kubernetes Resilience#361

Lab 12 submission - Advanced Kubernetes Resilience#361
N1qro wants to merge 38 commits into
inno-devops-labs:mainfrom
N1qro:feature/lab12

N1qro commented Jul 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

N1qro commented Jul 17, 2026

Lab 12 - Advanced Kubernetes Resilience

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants