feat: add backward compatibility review agent for component-developer plugin by ZdenekSrotyr · Pull Request #53 · keboola/ai-kit

ZdenekSrotyr · 2026-02-21T17:07:00Z

Summary

Adds a new @backward-compatibility-reviewer agent and review-backward-compatibility skill to the component-developer plugin. This agent reviews component PRs exclusively for backward compatibility (not code quality), checking whether changes could break existing user configurations, sync actions, or output tables.

The agent:

Extracts component IDs from .github/workflows/push.yml (supports single, multi-env-var, and matrix patterns)
Analyzes diffs for breaking change vectors across configSchema, Pydantic models, sync actions, output tables, state files, and Dockerfiles
Queries Keboola telemetry (MCP, project 133) for real-world impact (active configs, job stats, error rates)
Outputs a structured review with severity levels (HIGH/MEDIUM/LOW/SAFE) and verdict (APPROVE/REQUEST CHANGES/WARN)
Enforces anonymization — never writes client/project/stack identifiers into public PR comments

New files:

agents/backward-compatibility-reviewer.md — agent definition
skills/review-backward-compatibility/SKILL.md — full review procedure
skills/review-backward-compatibility/references/breaking-changes.md — detailed breaking change vectors
skills/review-backward-compatibility/references/telemetry-analysis.md — SQL queries and anonymization rules

Updated files: plugin.json (→ 3.1.0), marketplace.json (→ 1.7.0), both READMEs (docs + changelog)

Review & Testing Checklist for Human

Verify telemetry column name job_start_at — The kbc_job table column is job_start_at (VARCHAR), confirmed via live query against Project 133. Note: the existing telemetry-debugging.md in debug-component skill uses job_start_time which appears to be incorrect — consider fixing that file separately.
Verify the TO_VARCHAR(DATEADD(...)) date comparison pattern — Since job_start_at is a VARCHAR column (not TIMESTAMP), queries use TO_VARCHAR(DATEADD('day', -30, CURRENT_TIMESTAMP()), 'YYYY-MM-DD"T"HH24:MI:SS') for string comparison. This was validated live but the format string depends on the actual date format stored in the column.
Test the agent end-to-end on a real component PR — Invoke @backward-compatibility-reviewer on an actual component-* PR (ideally one with known breaking changes like a configSchema field removal) and verify the output is structured correctly, telemetry data appears, and no sensitive data leaks into the review comment.
Review anonymization coverage — Anonymization is enforced by prompt instructions only, not technical controls. Verify the instructions are strong enough to prevent accidental data leakage (project IDs, stack URLs, client names) in real-world use.
Verify internal project exclusion list — Currently only project 4214 is excluded. Confirm this is acceptable for initial release or provide additional internal project IDs to add.

Suggested test plan:

Install the plugin locally and run @backward-compatibility-reviewer against a component PR with a known configSchema change
Verify the structured output includes telemetry summary, breaking change assessment, and verdict
Confirm no project names, client names, or stack URLs appear in the generated review comment

Notes

This is a prompt-only change (markdown files) — no executable code to test in CI
The agent uses color: red which matches the README but note the existing reviewer.md agent uses color: purple (README says 🔴 Red for both — minor inconsistency)
No slash command equivalent (e.g., /compat-review) was added, unlike the existing code quality reviewer
Telemetry queries validated live against Project 133 for keboola.ex-sap-odata — Query 1 (active configs), Query 2 (job stats), Query 3 (config parameter usage), and Query 5 (stack distribution) all returned correct results

Updates since last revision

Fixed JSON indentation inconsistency in marketplace.json and plugin.json
REVERTED column name change: The actual column in kbc_job table is job_start_at (VARCHAR), not job_start_time. The previous commit incorrectly changed it based on Copilot suggestion referencing telemetry-debugging.md which itself has the wrong column name.
Fixed date comparison to use TO_VARCHAR(DATEADD(...)) pattern since job_start_at is VARCHAR, not TIMESTAMP
Validated all telemetry queries against live Project 133 data

Link to Devin run: https://app.devin.ai/sessions/19cd1740b214413bad4da6ff3716d181
Requested by: @ZdenekSrotyr

… plugin - New agent: backward-compatibility-reviewer (@backward-compatibility-reviewer) - New skill: review-backward-compatibility with SKILL.md - References: breaking-changes.md, telemetry-analysis.md - Telemetry-powered impact analysis via Keboola MCP - Structured review with severity levels and verdicts - Anonymized output - never exposes client/project identifiers - Bump component-developer plugin to 3.1.0 - Bump marketplace to 1.7.0 Co-Authored-By: unknown <>

devin-ai-integration · 2026-02-21T17:07:03Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

Co-Authored-By: unknown <>

Copilot

Pull request overview

This PR adds a new backward compatibility review agent to the component-developer plugin. The agent reviews Keboola component PRs exclusively for backward compatibility issues (not code quality), checking whether changes could break existing user configurations, sync actions, or output tables. It integrates with Keboola telemetry data via MCP to assess real-world impact and enforces strict anonymization rules for public repositories.

Changes:

Added @backward-compatibility-reviewer agent and review-backward-compatibility skill with comprehensive documentation
Integrated telemetry analysis via Keboola MCP (Project 133) for real-world impact assessment
Created detailed reference guides for breaking change vectors and SQL query patterns
Updated plugin to version 3.1.0 with corresponding marketplace and README changes

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`plugins/component-developer/agents/backward-compatibility-reviewer.md`	Agent definition with YAML frontmatter and overview of backward compatibility review capabilities
`plugins/component-developer/skills/review-backward-compatibility/SKILL.md`	Complete review procedure including component ID extraction, diff analysis, breaking change detection, telemetry queries, and structured review output
`plugins/component-developer/skills/review-backward-compatibility/references/breaking-changes.md`	Detailed guide covering all breaking change vectors across configSchema, Pydantic models, sync actions, output tables, Dockerfile, and state files
`plugins/component-developer/skills/review-backward-compatibility/references/telemetry-analysis.md`	SQL queries for telemetry data analysis with anonymization rules and workflow guidance
`plugins/component-developer/README.md`	Documentation update with new skill description, version bump to 3.1.0, and changelog entry
`plugins/component-developer/.claude-plugin/plugin.json`	Version bump to 3.1.0 and updated description
`.claude-plugin/marketplace.json`	Marketplace version update to 1.7.0 and component-developer plugin version to 3.1.0
`README.md`	Main repository README updated to list backward compatibility review in skills

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-21T17:34:27Z

...ns/component-developer/skills/review-backward-compatibility/references/telemetry-analysis.md

+  ) as error_rate_pct
+FROM "KBC_USE4_37"."out.c-kbc_public_telemetry"."kbc_job"
+WHERE "kbc_component_id" LIKE '<COMPONENT_ID>%'
+  AND "job_start_at" >= DATEADD('day', -30, CURRENT_TIMESTAMP())


The column name job_start_at appears to be incorrect based on the existing telemetry documentation. The telemetry-debugging.md file uses job_start_time for querying the kbc_job table. This should be updated to job_start_time to match the actual schema.

Suggested change

AND "job_start_at" >= DATEADD('day', -30, CURRENT_TIMESTAMP())

AND "job_start_time" >= DATEADD('day', -30, CURRENT_TIMESTAMP())

Fixed in 30dfd3f — changed job_start_at to job_start_time in both SKILL.md and telemetry-analysis.md to match the actual telemetry schema.

Copilot · 2026-02-21T17:34:27Z

plugins/component-developer/skills/review-backward-compatibility/SKILL.md

+  ROUND(SUM(CASE WHEN "job_status" = 'error' THEN 1 ELSE 0 END) * 100.0 / NULLIF(COUNT(*), 0), 1) as error_rate_pct
+FROM "KBC_USE4_37"."out.c-kbc_public_telemetry"."kbc_job"
+WHERE "kbc_component_id" LIKE '<COMPONENT_ID>%'
+  AND "job_start_at" >= DATEADD('day', -30, CURRENT_TIMESTAMP())


The column name job_start_at appears to be incorrect based on the existing telemetry documentation. The telemetry-debugging.md file uses job_start_time for querying the kbc_job table. This should be updated to job_start_time to match the actual schema.

Suggested change

AND "job_start_at" >= DATEADD('day', -30, CURRENT_TIMESTAMP())

AND "job_start_time" >= DATEADD('day', -30, CURRENT_TIMESTAMP())

…queries Co-Authored-By: unknown <>

The actual column in kbc_job table is job_start_at (VARCHAR), not job_start_time. The previous commit incorrectly changed it based on Copilot suggestion referencing telemetry-debugging.md which itself has the wrong column name. Also fix DATEADD comparison - since job_start_at is VARCHAR, use TO_VARCHAR to convert the timestamp for string comparison. Co-Authored-By: unknown <>

devin-ai-integration bot assigned ZdenekSrotyr Feb 21, 2026

fix: correct JSON indentation in marketplace.json and plugin.json

d1df098

Co-Authored-By: unknown <>

ZdenekSrotyr marked this pull request as ready for review February 21, 2026 17:28

ZdenekSrotyr requested a review from Copilot February 21, 2026 17:28

Copilot started reviewing on behalf of ZdenekSrotyr February 21, 2026 17:29 View session

Copilot AI reviewed Feb 21, 2026

View reviewed changes

devin-ai-integration bot added 2 commits February 21, 2026 17:57

fix: correct column name job_start_at to job_start_time in telemetry …

30dfd3f

…queries Co-Authored-By: unknown <>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

feat: add backward compatibility review agent for component-developer plugin#53

feat: add backward compatibility review agent for component-developer plugin#53
ZdenekSrotyr wants to merge 4 commits intomainfrom
devin/1771693293-backward-compatibility-reviewer

ZdenekSrotyr commented Feb 21, 2026 •

edited by devin-ai-integration bot

Loading

Uh oh!

devin-ai-integration bot commented Feb 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 21, 2026

Uh oh!

devin-ai-integration bot Feb 21, 2026

Uh oh!

Copilot AI Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	AND "job_start_at" >= DATEADD('day', -30, CURRENT_TIMESTAMP())
	AND "job_start_time" >= DATEADD('day', -30, CURRENT_TIMESTAMP())

Comments

Conversation

ZdenekSrotyr commented Feb 21, 2026 • edited by devin-ai-integration bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review & Testing Checklist for Human

Notes

Updates since last revision

Uh oh!

devin-ai-integration bot commented Feb 21, 2026

🤖 Devin AI Engineer

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 21, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Feb 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ZdenekSrotyr commented Feb 21, 2026 •

edited by devin-ai-integration bot

Loading