Pass ForwardOptions from top level module and also return any relevant state as output #8186

sxu · 2025-02-04T18:27:28Z

Summary: Pass a ForwardOptions argument (introduced by #8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. This is in preparation for a static shape attention implementation.

Differential Revision: D69080123

cc @mergennachin @cccclai @helunwencser @dvorjackz

pytorch-bot · 2025-02-04T18:27:33Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8186

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 867a885 with merge base dd31d93 ():

NEW FAILURE - The following job has failed:

pull / unittest / macos / macos-job (gh)
backends/xnnpack/test/ops/test_add.py::TestAdd::test_qs8_add_relu_seq

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-02-04T18:27:37Z

This pull request was exported from Phabricator. Differential Revision: D69080123

iseeyuan · 2025-02-04T18:37:17Z

@pytorchbot label "release notes: llama_transformer: Pass ForwardOptions from top level module and also return any relevant state as output"

pytorch-bot · 2025-02-04T18:37:22Z

Didn't find following labels among repository labels: release notes: llama_transformer: Pass ForwardOptions from top level module and also return any relevant state as output

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T19:19:47Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T21:10:01Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T21:59:42Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T22:22:08Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T22:49:15Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T23:04:02Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-04T23:17:20Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-05T04:06:08Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-05T04:50:26Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-05T16:27:42Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan Differential Revision: D69080123

facebook-github-bot · 2025-02-05T16:46:54Z

This pull request was exported from Phabricator. Differential Revision: D69080123

cccclai

Generally seems fine

cccclai · 2025-02-05T21:23:31Z

examples/models/llama/attention.py

@@ -236,7 +236,7 @@ def forward(
            assert input_pos is not None
            k, v = self.kv_cache.update(input_pos, k, v)
            output = self.SDPA(input_pos, q, k, v, bsz, seqlen, self.mask)
-            return self.wo(output)
+            return self.wo(output), None


Is None a place holder or something else?

It's to support additional output from each attention layer that'll be aggregated and returned by the top level transformer. To avoid disruptions if this is `None, the top level transformer still return a single logit output, otherwise it returns a tuple.

This was actually added in the previous PR that introduced the abstract attention class:

executorch/examples/models/llama/attention.py

Line 30 in 440a3ac

) -> Tuple[torch.Tensor, Optional[Any]]:

.

cccclai · 2025-02-05T21:25:33Z

examples/models/llama/model.py

-                    [0], dtype=torch.long
-                ),  # start_pos, what token of output are we on.
+                {
+                    "input_pos": torch.tensor(


Is it because we are plugging in the Forward option down the line?

Yeah, "input_pos" is the only thing the regular attention needs, other implementations can take additional/different parameters.

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan, cccclai Differential Revision: D69080123

facebook-github-bot · 2025-02-05T23:55:49Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan, cccclai Differential Revision: D69080123

facebook-github-bot · 2025-02-06T02:55:44Z

This pull request was exported from Phabricator. Differential Revision: D69080123

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan, cccclai Differential Revision: D69080123

facebook-github-bot · 2025-02-06T16:31:06Z

This pull request was exported from Phabricator. Differential Revision: D69080123

iseeyuan · 2025-02-06T17:26:13Z

Thanks @sxu ! Not sure how to connect to a discussion, but this PR is related to #8228

…t state as output (pytorch#8186) Summary: Pass a `ForwardOptions` argument (introduced by pytorch#8128) from the top level transformer, consolidate some existing inputs into it, and return any optional updates from the attention implementation. Reviewed By: iseeyuan, cccclai Differential Revision: D69080123

facebook-github-bot · 2025-02-06T19:03:19Z

This pull request was exported from Phabricator. Differential Revision: D69080123

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 4, 2025

facebook-github-bot added the fb-exported label Feb 4, 2025

sxu force-pushed the export-D69080123 branch from cbe146d to d9a5d21 Compare February 4, 2025 19:19

sxu force-pushed the export-D69080123 branch from d9a5d21 to aca7245 Compare February 4, 2025 21:09

sxu added the topic: not user facing label Feb 4, 2025

sxu force-pushed the export-D69080123 branch from aca7245 to d921670 Compare February 4, 2025 21:59

sxu force-pushed the export-D69080123 branch from d921670 to 045eef6 Compare February 4, 2025 22:22

sxu force-pushed the export-D69080123 branch from 045eef6 to 5f79682 Compare February 4, 2025 22:49

sxu force-pushed the export-D69080123 branch from 5f79682 to ec982a9 Compare February 4, 2025 23:03

sxu force-pushed the export-D69080123 branch from ec982a9 to ccf7082 Compare February 4, 2025 23:17

sxu force-pushed the export-D69080123 branch from ccf7082 to 04a6fb2 Compare February 5, 2025 00:25

sxu force-pushed the export-D69080123 branch from 4351436 to c514fa8 Compare February 5, 2025 04:50

sxu force-pushed the export-D69080123 branch from c514fa8 to b6e26b7 Compare February 5, 2025 16:27

sxu force-pushed the export-D69080123 branch from b6e26b7 to 8a78aba Compare February 5, 2025 16:46

sxu requested a review from cccclai February 5, 2025 18:52

cccclai approved these changes Feb 5, 2025

View reviewed changes

sxu force-pushed the export-D69080123 branch from 8a78aba to e25d751 Compare February 5, 2025 23:55

sxu force-pushed the export-D69080123 branch from e25d751 to ff88fe5 Compare February 6, 2025 02:55

sxu force-pushed the export-D69080123 branch from ff88fe5 to 39b9f86 Compare February 6, 2025 16:30

sxu force-pushed the export-D69080123 branch from 39b9f86 to 867a885 Compare February 6, 2025 19:03

facebook-github-bot merged commit b362ab7 into pytorch:main Feb 6, 2025
43 of 47 checks passed

This was referenced Feb 11, 2025

Weekly pr metrics report - 2025-02-01..2025-02-07 wdvr/pytorch#6

Open

Weekly pr metrics report - 2025-02-01..2025-02-07 wdvr/pytorch#8

Open

This was referenced Feb 24, 2025

Weekly pr metrics report - 2025-02-01..2025-02-07 wdvr/pytorch#10

Open

Weekly pr metrics report - 2025-02-01..2025-02-07 wdvr/pytorch#14

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pass ForwardOptions from top level module and also return any relevant state as output #8186

Pass ForwardOptions from top level module and also return any relevant state as output #8186

sxu commented Feb 4, 2025 •

edited

Loading

pytorch-bot bot commented Feb 4, 2025 •

edited

Loading

facebook-github-bot commented Feb 4, 2025

iseeyuan commented Feb 4, 2025

pytorch-bot bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

cccclai left a comment

cccclai Feb 5, 2025

sxu Feb 5, 2025

cccclai Feb 5, 2025

sxu Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 6, 2025

facebook-github-bot commented Feb 6, 2025

iseeyuan commented Feb 6, 2025

facebook-github-bot commented Feb 6, 2025

Pass ForwardOptions from top level module and also return any relevant state as output #8186

Pass ForwardOptions from top level module and also return any relevant state as output #8186

Conversation

sxu commented Feb 4, 2025 • edited Loading

pytorch-bot bot commented Feb 4, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8186

❌ 1 New Failure

facebook-github-bot commented Feb 4, 2025

iseeyuan commented Feb 4, 2025

pytorch-bot bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 4, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 5, 2025

cccclai left a comment

Choose a reason for hiding this comment

cccclai Feb 5, 2025

Choose a reason for hiding this comment

sxu Feb 5, 2025

Choose a reason for hiding this comment

cccclai Feb 5, 2025

Choose a reason for hiding this comment

sxu Feb 5, 2025

Choose a reason for hiding this comment

facebook-github-bot commented Feb 5, 2025

facebook-github-bot commented Feb 6, 2025

facebook-github-bot commented Feb 6, 2025

iseeyuan commented Feb 6, 2025

facebook-github-bot commented Feb 6, 2025

sxu commented Feb 4, 2025 •

edited

Loading

pytorch-bot bot commented Feb 4, 2025 •

edited

Loading