v5.0 Inference post-mortem item - Enforcing rules on late submissions #314
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
|
||
| **No modifications** to hyperparameters, batch size, loadgen configurations and flags, dataset sorting, or any other performance-sensitive parameters are allowed after the submission deadline. This restriction applies even if such changes are necessary to comply with the rules. | ||
|
|
||
| Under no circumstances will submitters be allowed to upload new logs that alter the accuracy, performance, or power results, regardless of the extent of the change. |
There was a problem hiding this comment.
Can we add a 1-2% delta here to account for noise - if the change is asked by the review committee?
| @@ -129,6 +129,21 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm | |||
|
|
|||
| An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group. | |||
|
|
|||
| === Late Submissions and Late Upload of Logs | |||
|
|
|||
| To ensure fairness to all submitters and the integrity of the process, MLPerf Inference will strictly adhere to the https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#late-submissions[MLCommons general submission rules]. The 60 minute grace period will be strictly enforced, and any late submissions will not be considered as valid submissions. Exceptions will be made only in extraordinary circumstances as listed in the https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#post-submission-extension-for-extraordinary-circumstances-submission-deadline-72-hours[mlcommons guidelines]. | |||
There was a problem hiding this comment.
This PR is much appreciated considering the amount of time which is spent during the review meetings on such requests. The one concern though is even though this PR is making the rules strict it is not preventing any such late submission requests from coming to the review committee. Also we know that the rules are a bit relaxed for first time submitters - but "how relaxed" is not defined. It would be good if the rules are clear so that the review committee chair can show that to the submitters asking for allowing any late submissions instead of those requests coming to the review committee for discussions.
|
|
||
| **Exceptions** to this rule may be granted solely at the discretion of the Review Committee, and only to ensure compliance with the established rules. For instance, if logs are corrupted during the submission process or if the submission checker fails to detect missing logs, the Review Committee may authorize a re-upload of the affected logs. | ||
|
|
||
| **No modifications** to hyperparameters, batch size, loadgen configurations and flags, dataset sorting, or any other performance-sensitive parameters are allowed after the submission deadline. This restriction applies even if such changes are necessary to comply with the rules. |
There was a problem hiding this comment.
Should we explicitly state that modifications are indeed allowed in case of valid hyperparameter borrowing scenarios which is actually allowed in Training during the review period?
There was a problem hiding this comment.
Should we explicitly state that modifications are indeed allowed in case of valid hyperparameter borrowing scenarios which is actually allowed in Training during the review period?
This repo has the policy applies to Inference only, and we don’t have any HP borrowing in Inference.
This PR addresses issues that arose in the v5.0 and prior inference reviews and attempts to clarify the rules surrounding late submissions.
This PR:
The goal is: