Repository navigation
Add the analyze_async_overlap CLI subcommand to measure how much of each asynchronous XLA operation (collectives, async copies, send/recv) is hidden behind compute on the same device, and how much time multiple engines on the same TPU chip (TensorCore and SparseCore) spend busy concurrently: - #3830
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
… each asynchronous XLA operation (collectives, async copies, `send`/`recv`) is hidden behind compute on the same device, and how much time multiple engines on the same TPU chip (TensorCore and SparseCore) spend busy concurrently:
* `cli/internal/async_overlap.py`: Pure-Python interval analysis over device `TimelineEvent`s (`XLA Ops`, `Async XLA Ops`, `Sparse Core Ops`). Uses `Async XLA Ops` windows when present and falls back to FIFO `<op>-start` / `<op>-done` pairing on the op lines when absent. Classifies each async window as `hidden` (covered fraction >= `hidden_threshold`, default 0.9), `exposed` (<= `exposed_threshold`, default 0.1), or `partial`, and computes pairwise concurrent busy time across engines per TPU chip.
* `cli/internal/{google,oss}/xplane_tools.py`: Adds `iter_timeline_events` to extract device timeline events with absolute picosecond start times across both XPlane backends.
* `cli/tools/analyze_async_overlap_tool.py`: CLI/MCP tool wrapper registered as `analyze_async_overlap` in `xprof_cli.py` and `server.py`.
* `skills/xprof/`: Documents `analyze_async_overlap` in `references/analyze_async_overlap.md` and `SKILL.md`.
PiperOrigin-RevId: 997089732
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the
analyze_async_overlapCLI subcommand to measure how much of each asynchronous XLA operation (collectives, async copies,send/recv) is hidden behind compute on the same device, and how much time multiple engines on the same TPU chip (TensorCore and SparseCore) spend busy concurrently:cli/internal/async_overlap.py: Pure-Python interval analysis over deviceTimelineEvents (XLA Ops,Async XLA Ops,Sparse Core Ops). UsesAsync XLA Opswindows when present and falls back to FIFO<op>-start/<op>-donepairing on the op lines when absent. Classifies each async window ashidden(covered fraction >=hidden_threshold, default 0.9),exposed(<=exposed_threshold, default 0.1), orpartial, and computes pairwise concurrent busy time across engines per TPU chip.cli/internal/{google,oss}/xplane_tools.py: Addsiter_timeline_eventsto extract device timeline events with absolute picosecond start times across both XPlane backends.cli/tools/analyze_async_overlap_tool.py: CLI/MCP tool wrapper registered asanalyze_async_overlapinxprof_cli.pyandserver.py.skills/xprof/: Documentsanalyze_async_overlapinreferences/analyze_async_overlap.mdandSKILL.md.