-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Fix CLI validation of summary trend stats #5425
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Fix CLI validation of summary trend stats #5425
Conversation
ankur22
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @weillercarvalho thanks for contributing a fix!
I wanted to understand the original issue, so as you suggested in the original issue, I:
- Created an empty test script using
export default function () {}. - Ran the test with
k6 run empty.js --summary-trend-stats="med,avg,p(101)"
However the test didn't run and I was presented with:
ERRO[0000] invalid percentile trend stat value 'p(101)', provide a number between 0 and 100
exit status 255After applying the fix (which does indeed look correct) I get the same result though. What were you seeing before the fix?
|
Here is a quick explanation: Before the patch, the CLI flag parser (getOptions()) tried to validate --summary-trend-stats, but it checked the wrong error variable. So even clearly bad values like p(101) slipped through that first gate and only failed later, when k6 run reached getConsolidatedConfig() and the config consolidation stage re-ran the same validation. That’s why @ankur22 you already saw ERRO[0000] invalid percentile… before my change: the later stage has always rejected invalid stats. The fix just moves the failure to the place it should have always happened: as soon as the flag is parsed. We now return the validation error directly from getOptions(), and the consolidation tests were updated to expect a CLI read error for malformed stats. So the final user-facing error message didn’t change; what changed is that the CLI no longer accepts bogus stats for a moment before rejecting them. This keeps the behavior consistent across all commands that call getOptions(), not only the ones that happen to run the full consolidation pipeline. |
ankur22
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for explaining @weillercarvalho! LGTM 🚀
codebien
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@weillercarvalho That's a good catch, thanks for contributing! 🙇 LGTM, I'll merge as soon as we have a green CI.
What?
Why?
Users could pass malformed stats (e.g. p(101), coun, p() without any feedback; the CLI would accept them and the test would run
even though the stats were invalid. Surfacing the error when parsing flags prevents confusing, silent failures.
Checklist
browser/DNS-dependent suites in this environment.)
Checklist: Documentation (only for k6 maintainers and if relevant)
applicable
types/k6): grafana/k6-DefinitelyTyped#NUMBER if applicable
Related PR(s)/Issue(s)
#5424