No such file or directory: executorch/exir/_serialize/program.fbs and scalar_type.fbs

[adonnini]

Hi I installed executorch from the main branch.

When I ran code to produce and store a lowered mode, execution failed with the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/program.fbs'

I checked older versions of executorch installed from main, e.g. on 3/18/2024, program.fbs was in executorch/exir/_serialize/

Please find the complete traCeback log below.

Did I make a mistake?

Please let me know if you need additional information and need me to do anything.

Thanks

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 460, in <module>
    f.write(lowered_module.to_executorch().buffer)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/program/_program.py", line 871, in to_executorch
    return ExecutorchProgramManager(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/program/_program.py", line 925, in __init__
    self._pte_data: Cord = _serialize_pte_binary(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_program.py", line 424, in serialize_pte_binary
    result: _FlatbufferResult = _program_json_to_flatbuffer(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_flatbuffer.py", line 283, in _program_json_to_flatbuffer
    schema_info = _prepare_schema(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_flatbuffer.py", line 156, in _prepare_schema
    schemas = _ResourceFiles([program_schema] + deps)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_flatbuffer.py", line 113, in __init__
    self._files[name] = importlib.resources.read_binary(__package__, name)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/importlib/resources.py", line 88, in read_binary
    with open_binary(package, resource) as fp:
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/importlib/resources.py", line 46, in open_binary
    return reader.open_resource(resource)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/importlib/abc.py", line 433, in open_resource
    return self.files().joinpath(resource).open('rb')
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/pathlib.py", line 1117, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/program.fbs'

[orionr]

cc @guangy10 @dbort @byjlw @mergennachin

[adonnini]

Btw, Copying program.fbs and scalar_type.fbs in exir/_serializer solved my problem. I know it's not a solution but it tells me that's it's probably not a big deal to resolve.

[kirklandsign]

Try unset PYTHONPATH?

I ran into this as well. If I set PYTHONPATH to executorch root directory, it first searches for executorch/exir/_serializer before it searches for the pip package for executorch. We only copy program.fbs and scalar_type.fbs to pip package, not executorch source package.

[kirklandsign]

That seems to be introduced in #2499. @dbort Maybe there can be compatibility concern like this?

[dbort]

Yes, the problem here seems to be that python is executing the .py files under your executorch directory instead of using the files installed by the pip package.

The old pip installation step incorrectly copied the .fbs files into the source tree; the new pip installation builds the package hermetically without modifying the source tree.

@adonnini what command are you running when this happens? Also, are you running that command from a directory that contains the executorch repo/directory?

[adonnini]

@dbort I follow the instructions in
https://pytorch.org/executorch/stable/getting-started-setup.html (running ./install_requirements.sh)
in the executorch directory.
I have done this many times both installing the v0.1.0 and from the main branch. This was the first time this issue happened.
I will be doing this again around the middle of next week.
Thanks

[dbort]

@adonnini That is the correct way to install the pip package. But the problems you're seeing are probably related to how you're running the failing program.

Earlier you said

When I ran code to produce and store a lowered mode, execution failed with the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/program.fbs'

What command/code are you running when this error happens? What is your current directory when you run this command?

[adonnini]

@dbort , sorry I misunderstood your question. Please find below the code I run to produce the lowered model, which is the code I executed when the problem occurred.
Please let me know if you need anything else.
Thanks

CODE TO LOWER MODEL

#Delegating to a Backend - Lowering the Whole Module
#---------------------------------------------------
# Export and lower the module to Edge Dialect
pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))
aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask), strict=False)
edge_program: EdgeProgramManager = to_edge(aten_dialect)
to_be_lowered_module = edge_program.exported_program()

from executorch.exir.backend.backend_api import LoweredBackendModule, to_backend

# Lower the module
lowered_module = edge_program.to_backend(XnnpackPartitioner())

print(" - train_minimum - Lowering the Whole Module - lowered_module - ", lowered_module)

# Serialize and save it to a file
save_path = save_path = "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tpt_delegate.pte"
with open(save_path, "wb") as f:
    f.write(lowered_module.to_executorch().buffer)

[dbort]

Thanks for that, @adonnini.

When you run this command, what is your current directory? Does that directory contain a directory named executorch, which contains the ExecuTorch repo?

[adonnini]

I run the code from a directory which contains the executorch directory. It contains the Executorch repo (including exir)
Does this answer your questions?

[metascroy]

@adonnini can you try running your code from a directory that does not contain the cloned executorch repo?

[adonnini]

@metascroy, sorry for the delay in responding. I ran the code as you suggested.
The No such file or directory error did not occur.
Please let me know if I need to do anything else

[dbort]

I'm glad that this worked. It is a pain that things behave differently based on the contents of the current directory; that's something that we hope to fix later this year.