[Request impl] Devtool end-to-end tests

[Gasoonjia]

🚀 The feature, motivation and pitch

Devtools now have multiple components, like etdump, etrecrod, inspector, etc. Inpsector should take etdump and etrecord (optional) as input, but now we only have tests for each module separately, but do not have end-to-end tests, which may introduce some risk for our debugging pipeline. We would love for you to help us build end-to-end testing.

RFC

The overall picture of the e2e test will be like:

1. Export a simple module (e.g. an module with only add or multiple operation) and dump into buffer; using etrecord to record information; see here for example but maybe use current export flow
2. Using pybinding to load the buffer and equip with etdump
3. Propagate the etrecord and etdump to inspector api ; choose one or two inspector functions (e.g. to_dataframe) and compare the result with your expectation.

This doc might be a good resource for you to understand the end2end pipeline better.

[HonestDeng]

It's an interesting and helpful issue for me to start. Would it be alright if I take this?

[Gasoonjia]

Sure!

[HonestDeng]

Hey bro! I've seen a directory named directory/test_end2end.py.

I was thinking of creating a new file in this directory to add a unittest class. Within this class, I plan to write methods to create etdump, etrecord, and inspector. Does this approach sound reasonable?

[Gasoonjia]

@HonestDeng Thanks for your response!

directory/test_end2end.py

Which directory here are you working on? I think executorch/devtools/test/test_end2end.py would be great.

Does this approach sound reasonable?

Yeah as long as they will be composed into a single e2e test pipeline that should be fine. Target of e2e test will be testing the pipeline as a whole, rather than testing them as individual module

[HonestDeng]

Oh no, my bad! I mistakenly wrote directory/test_end2end.py. I actually meant executorch/test/end2end.

I'll implement it in executorch/devtools/test/test_end2end.py. Thanks for your response!

[HonestDeng]

Based on my reading of the documentation, here is my proposed approach:

1. Model: Prepare a simple nn.Module.
2. ETRecord: Generate the etrecord using the Python code provided in the docs.
3. ETDump: According to the docs, a bundled_program is needed to generate the etdump. The etdump itself is generated by C++ code (e.g., examples/devtools/example_runner.cpp). Therefore, after generating the bundled_program in Python, a C++ file with Python bindings (pybinding) is required. This allows the Python code to invoke the C++ function to generate the etdump.
4. Inspector: Generate the Inspector artifact using the Python code specified in the docs.

Finally, assertions, such as self.assertEqual, are needed to verify that the test executed successfully.

[Gasoonjia]

Thanks for sharing! The overall logic looks great!
For the ETDump step, you can directly use our pybinding api to generate etdump: https://github.com/pytorch/executorch/blob/main/extension/pybindings/pybindings.pyi#L151

[HonestDeng]

Hey team, I've successfully generated the etrecord and bundled_program. However, I'm having trouble figuring out how to generate the etdump.

I looked at the _load_for_executorch_from_bundled_program function in pybindings.pyi (link) and tried setting enable_etdump=True. But I'm confused – where does the etdump file get generated or saved when using this approach?

I also reviewed the C++ code in examples/devtools/example_runner.cpp (link). This example seems to imply that I need to actually execute the module to generate the etdump.

If execution is indeed required, how can I achieve this using the Python bindings? Specifically, how do I run the loaded module via Python in a way that triggers the etdump generation?

Thanks a lot!

[Gasoonjia]

you can use write_etdump_result_to_file function https://github.com/pytorch/executorch/blob/main/extension/pybindings/pybindings.cpp#L1119 to generate etdump file as well as its debug buffer file to the address you want.

[HonestDeng]

I build the project using

cmake .. -DEXECUTORCH_BUILD_DEVTOOLS=ON -DET_EVENT_TRACER_ENABLED=ON

Then try to run:

python devtools/test/test_end2end.py

But got:

[program.cpp:136] InternalConsistency verification requested but not available [pybindings.cpp:821] No etdump data found, try rebuilding with the CMake option EXECUTORCH_ENABLE_EVENT_TRACER or with buck run --config executorch.event_tracer_enabled=true

I also try to use:

cmake .. -DEXECUTORCH_BUILD_DEVTOOLS=ON -DEXECUTORCH_ENABLE_EVENT_TRACER=ON

But still failed.

why? Here is the commit e449350