[ET-VK][AOT] Serialize constant tensors via NamedDataMap #13473

SS-JIA · 2025-08-18T15:16:29Z

Stack from ghstack (oldest at bottom):

Summary:
When exporting models to Vulkan backend, save constant tensors in the NamedDataMap instead of the constant data section of the delegate header.

Motivation

Prevent screen blackout (Llama 3.2 1B) / device crash (Llama 3.2 3B) when running Llama 3.2 models on Samsung Galaxy S24. This behaviour is related to high peak memory usage when loading the model. For more information, see the top diff/PR in the stack.

Context

This change is based on the equivalent change D70315207/#9153 in XNNPACK.

Test Plan:

Memory Comparison with/without NamedDataMap

Measured VmRss using

uint64_t getVmRssInKB() {
  std::ifstream statusFile("/proc/self/status");
  std::string l, num;
  while (std::getline(statusFile, l)) {
    if (l.substr(0, 5) == "VmRSS") {
      size_t pos = l.find_first_of("0123456789");
      num = l.substr(pos);
      break;
    }
  }
  uint64_t vmRssInKB = std::stoi(num);
  return vmRssInKB;
}

P1908019767 (Meta only)

Excerpt:

Log 1                                             | Log 2
--------------------------------------------------|--------------------------------------------------
Memory usage before model compilation: 1115416 KB | Memory usage before model compilation: 1919228 KB
Memory usage after graph building: 1924340 KB     | Memory usage after graph building: 1924256 KB
Memory usage after graph preparation: 1798968 KB  | Memory usage after graph preparation: 1782464 KB
Memory usage prepack start: 1798968 KB            | Memory usage prepack start: 1781968 KB

Memory usage after prepack operations: 1271924 KB | Memory usage after prepack operations: 1653496 KB

Differential Revision: D80460034

Summary: When exporting models to Vulkan backend, save constant tensors in the NamedDataMap instead of the constant data section of the delegate header. ## Motivation Prevent screen blackout (Llama 3.2 1B) / device crash (Llama 3.2 3B) when running Llama 3.2 models on Samsung Galaxy S24. This behaviour is related to high peak memory usage when loading the model. For more information, see the top diff/PR in the stack. ## Context This change is based on the equivalent change D70315207/#9153 in XNNPACK. Test Plan: ## Memory Comparison with/without NamedDataMap Measured VmRss using ``` uint64_t getVmRssInKB() { std::ifstream statusFile("/proc/self/status"); std::string l, num; while (std::getline(statusFile, l)) { if (l.substr(0, 5) == "VmRSS") { size_t pos = l.find_first_of("0123456789"); num = l.substr(pos); break; } } uint64_t vmRssInKB = std::stoi(num); return vmRssInKB; } ``` P1908019767 (Meta only) Excerpt: ``` Log 1 | Log 2 --------------------------------------------------|-------------------------------------------------- Memory usage before model compilation: 1115416 KB | Memory usage before model compilation: 1919228 KB Memory usage after graph building: 1924340 KB | Memory usage after graph building: 1924256 KB Memory usage after graph preparation: 1798968 KB | Memory usage after graph preparation: 1782464 KB Memory usage prepack start: 1798968 KB | Memory usage prepack start: 1781968 KB Memory usage after prepack operations: 1271924 KB | Memory usage after prepack operations: 1653496 KB ``` [ghstack-poisoned]

pytorch-bot · 2025-08-18T15:16:32Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13473

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 13b671d with merge base 8ef9595 ():

NEW FAILURE - The following job has failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / test-binary-size-linux-gcc / linux-job (gh) (trunk failure)
/pytorch/executorch/kernels/portable/cpu/op_stack.cpp:129:26: error: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘ssize_t’ {aka ‘long int’} [-Werror=sign-compare]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-08-18T15:17:13Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

SS-JIA · 2025-08-18T15:25:42Z

@SS-JIA has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: When exporting models to Vulkan backend, save constant tensors in the NamedDataMap instead of the constant data section of the delegate header. ## Motivation Prevent screen blackout (Llama 3.2 1B) / device crash (Llama 3.2 3B) when running Llama 3.2 models on Samsung Galaxy S24. This behaviour is related to high peak memory usage when loading the model. For more information, see the top diff/PR in the stack. ## Context This change is based on the equivalent change D70315207/#9153 in XNNPACK. Test Plan: ## Memory Comparison with/without NamedDataMap Measured VmRss using ``` uint64_t getVmRssInKB() { std::ifstream statusFile("/proc/self/status"); std::string l, num; while (std::getline(statusFile, l)) { if (l.substr(0, 5) == "VmRSS") { size_t pos = l.find_first_of("0123456789"); num = l.substr(pos); break; } } uint64_t vmRssInKB = std::stoi(num); return vmRssInKB; } ``` P1908019767 (Meta only) Excerpt: ``` Log 1 | Log 2 --------------------------------------------------|-------------------------------------------------- Memory usage before model compilation: 1115416 KB | Memory usage before model compilation: 1919228 KB Memory usage after graph building: 1924340 KB | Memory usage after graph building: 1924256 KB Memory usage after graph preparation: 1798968 KB | Memory usage after graph preparation: 1782464 KB Memory usage prepack start: 1798968 KB | Memory usage prepack start: 1781968 KB Memory usage after prepack operations: 1271924 KB | Memory usage after prepack operations: 1653496 KB ``` Differential Revision: [D80460034](https://our.internmc.facebook.com/intern/diff/D80460034) [ghstack-poisoned]

facebook-github-bot · 2025-08-18T17:28:55Z

This pull request was exported from Phabricator. Differential Revision: D80460034

Summary: When exporting models to Vulkan backend, save constant tensors in the NamedDataMap instead of the constant data section of the delegate header. ## Motivation Prevent screen blackout (Llama 3.2 1B) / device crash (Llama 3.2 3B) when running Llama 3.2 models on Samsung Galaxy S24. This behaviour is related to high peak memory usage when loading the model. For more information, see the top diff/PR in the stack. ## Context This change is based on the equivalent change D70315207/#9153 in XNNPACK. Test Plan: ## Memory Comparison with/without NamedDataMap Measured VmRss using ``` uint64_t getVmRssInKB() { std::ifstream statusFile("/proc/self/status"); std::string l, num; while (std::getline(statusFile, l)) { if (l.substr(0, 5) == "VmRSS") { size_t pos = l.find_first_of("0123456789"); num = l.substr(pos); break; } } uint64_t vmRssInKB = std::stoi(num); return vmRssInKB; } ``` P1908019767 (Meta only) Excerpt: ``` Log 1 | Log 2 --------------------------------------------------|-------------------------------------------------- Memory usage before model compilation: 1115416 KB | Memory usage before model compilation: 1919228 KB Memory usage after graph building: 1924340 KB | Memory usage after graph building: 1924256 KB Memory usage after graph preparation: 1798968 KB | Memory usage after graph preparation: 1782464 KB Memory usage prepack start: 1798968 KB | Memory usage prepack start: 1781968 KB Memory usage after prepack operations: 1271924 KB | Memory usage after prepack operations: 1653496 KB ``` Differential Revision: [D80460034](https://our.internmc.facebook.com/intern/diff/D80460034) [ghstack-poisoned]

facebook-github-bot · 2025-08-18T20:15:18Z

This pull request was exported from Phabricator. Differential Revision: D80460034

digantdesai · 2025-08-19T12:15:22Z

backends/vulkan/serialization/vulkan_graph_builder.py

+                sha256_hash = hashlib.sha256(tensor_bytes)
+                named_key = sha256_hash.hexdigest()
+
+                # Add to named data store with 16-byte alignment (matching XNNPACK)


digantdesai · 2025-08-19T12:16:22Z

backends/vulkan/serialization/vulkan_graph_builder.py

+                # Generate SHA256 hash as the named key
+                tensor_bytes = bytes(array)
+                sha256_hash = hashlib.sha256(tensor_bytes)
+                named_key = sha256_hash.hexdigest()


we deduplicate the data in the pte using the value, and not the key right?

This was referenced Aug 18, 2025

[ET-VK] Runtime support for NamedDataMap #13472

Merged

[ET-VK] Allocate memory for weight and activation tensors lazily #13474

Merged

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 18, 2025

SS-JIA mentioned this pull request Aug 18, 2025

[ET-VK][ez] Move execute node threshold calculation from prepare_pipelines() to prepare() #13478

Merged

facebook-github-bot added the fb-exported label Aug 18, 2025

lucylq approved these changes Aug 18, 2025

View reviewed changes

facebook-github-bot merged commit 546ec43 into gh/SS-JIA/291/base Aug 19, 2025
102 of 106 checks passed

facebook-github-bot deleted the gh/SS-JIA/291/head branch August 19, 2025 02:24

facebook-github-bot temporarily deployed to cherry-pick-bot August 19, 2025 02:24 — with GitHub Actions Inactive

pytorchbot mentioned this pull request Aug 19, 2025

[ET-VK][AOT] Serialize constant tensors via NamedDataMap #13499

Merged

digantdesai reviewed Aug 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ET-VK][AOT] Serialize constant tensors via NamedDataMap #13473

[ET-VK][AOT] Serialize constant tensors via NamedDataMap #13473

Uh oh!

SS-JIA commented Aug 18, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 18, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 18, 2025

Uh oh!

SS-JIA commented Aug 18, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

Uh oh!

digantdesai Aug 19, 2025

Uh oh!

digantdesai Aug 19, 2025

Uh oh!

Uh oh!

[ET-VK][AOT] Serialize constant tensors via NamedDataMap #13473

[ET-VK][AOT] Serialize constant tensors via NamedDataMap #13473

Uh oh!

Conversation

SS-JIA commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Context

Memory Comparison with/without NamedDataMap

Uh oh!

pytorch-bot bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13473

❌ 1 New Failure, 1 Unrelated Failure

Uh oh!

github-actions bot commented Aug 18, 2025

This PR needs a release notes: label

Uh oh!

SS-JIA commented Aug 18, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

Uh oh!

digantdesai Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

digantdesai Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SS-JIA commented Aug 18, 2025 •

edited

Loading

pytorch-bot bot commented Aug 18, 2025 •

edited

Loading

This PR needs a `release notes:` label