[ExecuTorch][to_backend] Enable to_backend API to leverage preprocess_all

[mcr229]

Stack from ghstack (oldest at bottom):

• -> [ExecuTorch][to_backend] Enable to_backend API to leverage preprocess_all #9811
• [ExecuTorch][to_backend] Introduce preprocess_all method to backend details #9810

We add a new to_backend api which essentially takes in:

Method --> ExportedProgram
Method --> Partitioner

This new to_backend api will then return

Method --> ExportedProgram

in which the ExportedPrograms are lowered using the partitioner. The key difference from the other to_backend implementation is that this implementation leverages preprocess_all.

The existing implementation goes in the following steps:

1. Partition the ExportedProgram and return a tagged module, each tag represents a partition, and nodes belonging to a partition have the tag in its metadata
2. loop through every tag(partition)
   2.1. create a submodule and call submodule node for the partition
   2.2. create an exported program from the submodule
   2.3. lower this exported program to the specified backend_id and generate a loweredbackendmodule
   2.4. replace the call submodule node with call delegate node and the generated loweredbackend module
   2.5 adjust the original owning program.
3. Return the newly adjusted owning program

The new implementation reorders the steps so that preprocess_all can be given all the partitioned programs at once:

1. loop through every method with a specified partitioner
   2. Partition the method's corresponding ExportedProgram with the specified partitioner and return a tagged module, each tag represents a partition, and nodes belonging to a partition have the tag in its metadata
   3. loop through every tag(partition)
   4. create a submodule and call submodule node for the partition
   5. create an exported program from the submodule
   6. store a bunch of meta data like owning_graph, is_submodule, delegation_spec, exported_program in the call_submodule's metadata
   7. return the list of all submodule nodes created for each partition
   8. Add mapping of method to the list of partitioned submodule nodes
2. Sort the above mapping by backend type (backend_id)
3. loop through backend types:
   11. lower all the submodule nodes of this backend type at once with preprocess_all
   12. loop through lowered call submodule nodes and replace them with results from preprocess_all
4. Return mapping of method to lowered programs

The new to_backend api is not used any where, the intent is for it to eventually be used in the EdgeProgramManager's to_backend implementation which already has and uses method_to_partitioner and method_to_program mappings.

Differential Revision: D69954542

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9811

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 75 Pending

As of commit b5e9996 with merge base d023a77 ():

NEW FAILURE - The following job has failed:

• Check Labels / Check labels (gh)
  RuntimeError: Error checking labels: PR does not have required labels

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D69954542

[github-actions]

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[mcr229]

Abandoning