Add support for missing tasks in mtgp #2960

sdaulton · 2025-08-08T17:57:29Z

Summary:
Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data.

This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair.

This will still error out in Ax if data for the target trial is missing.

Differential Revision: D79812024

facebook-github-bot · 2025-08-08T17:57:38Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

codecov · 2025-08-08T18:08:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (cc781b0) to head (33aceb7).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main     #2960   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          216       216           
  Lines        20328     20338   +10     
=========================================
+ Hits         20328     20338   +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Summary: X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-08T18:53:56Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Pull Request resolved: meta-pytorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: Pull Request resolved: facebook#4121 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-08T22:55:02Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-08T23:05:29Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-09T06:01:35Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-15T18:44:48Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Pull Request resolved: meta-pytorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 Pull Request resolved: facebook#4121 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-18T09:09:31Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

facebook-github-bot · 2025-08-20T12:40:13Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

facebook-github-bot · 2025-08-20T15:52:37Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Pull Request resolved: meta-pytorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

facebook-github-bot · 2025-08-20T19:12:09Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 Pull Request resolved: facebook#4121 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 Pull Request resolved: #4121 X-link: meta-pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Reviewed By: ItsMrLin Differential Revision: D79812024 fbshipit-source-id: f92a49fb46222698fdea13b5d96841bf2d5d679b

facebook-github-bot · 2025-08-21T15:53:08Z

This pull request has been merged in 9058a77.

facebook-github-bot · 2025-09-05T17:56:02Z

This pull request has been reverted by 3135a7b.

meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Aug 8, 2025

facebook-github-bot added the fb-exported label Aug 8, 2025

sdaulton mentioned this pull request Aug 8, 2025

Add support for missing tasks in mtgp facebook/Ax#4121

Closed

sdaulton force-pushed the export-D79812024 branch from f03dda4 to fae6912 Compare August 8, 2025 18:50

sdaulton force-pushed the export-D79812024 branch from fae6912 to 3ad186d Compare August 8, 2025 18:54

sdaulton force-pushed the export-D79812024 branch from 3ad186d to 5ef3c77 Compare August 8, 2025 22:54

sdaulton force-pushed the export-D79812024 branch from 5ef3c77 to 023b232 Compare August 8, 2025 23:05

sdaulton force-pushed the export-D79812024 branch from 023b232 to 6c977ee Compare August 9, 2025 06:01

sdaulton force-pushed the export-D79812024 branch from 6c977ee to fc0e7fd Compare August 12, 2025 06:33

sdaulton force-pushed the export-D79812024 branch from 589ff95 to b51fc3c Compare August 15, 2025 18:44

sdaulton force-pushed the export-D79812024 branch from b51fc3c to 2470005 Compare August 18, 2025 09:09

sdaulton force-pushed the export-D79812024 branch from 2470005 to e047ee5 Compare August 20, 2025 12:40

sdaulton force-pushed the export-D79812024 branch from e047ee5 to f0beefb Compare August 20, 2025 15:52

sdaulton force-pushed the export-D79812024 branch from f0beefb to 63a0f64 Compare August 20, 2025 19:08

sdaulton force-pushed the export-D79812024 branch from 63a0f64 to 33aceb7 Compare August 20, 2025 19:12

facebook-github-bot closed this in 9058a77 Aug 21, 2025

facebook-github-bot added the Merged label Aug 21, 2025

facebook-github-bot added the Reverted label Sep 5, 2025

Add support for missing tasks in mtgp #2960

Add support for missing tasks in mtgp #2960

Uh oh!

Conversation

sdaulton commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

codecov bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 9, 2025

Uh oh!

facebook-github-bot commented Aug 15, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

facebook-github-bot commented Aug 20, 2025

Uh oh!

facebook-github-bot commented Aug 20, 2025

Uh oh!

facebook-github-bot commented Aug 20, 2025

Uh oh!

facebook-github-bot commented Aug 21, 2025

Uh oh!

facebook-github-bot commented Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Aug 8, 2025 •

edited

Loading