Description
URL(s) with the issue:
Description of issue (what needs changing):
The current notebook runs without issue and works as a starting point. For me (and I presume others) the next step is naturally to organize the code in a more production-like pipeline which means adapting the notebook and fitting it into something like the templates described in this guide.
However, if one adapts the recommender tutorial to run outside of InteractiveContext, then the code fails to run. In particular, the Channel's that we pass to the Trainer component are empty when the pipeline is run using LocalDagRunner. When the MovielensModel calls movies_uri.get()[0] in its constructor, the program will throw a RuntimeError because we are indexing into an empty list. This is in spite of the fact that the artifacts do exist in the local file system and previous components have run correctly.
I created a fork and (arbitrarily) pushed my code here to illustrate exactly what I am running.
You can see the logs from a run here. In particular, look from this line onwards and you will see what custom_config evaluates to and the error.
From my brief attempt at tracing through the TFX code for the Trainer component, it seems that the executors track or resolve the artifacts for the non-custom_config arguments (like examples, transform_graph, and schema) differently than the custom_config arguments. That is why train_files in run_fn() is a valid path while the custom_config values are empty Channel's. But I am uncertain of why there is a difference depending on the orchestrator used and what the correct way to resolve this is.
This is not a new confusion, as you can see others have come across the same situation as myself. Unfortunately, that question was never answered and I was also unable to find any answers in any of the TensorFlow repos/docs or other stack overflow posts. I hope that this issue can clarify the correct way to approach this situation and help others avoid the same mistake in the future.
Why this should be changed:
I would like to request that the tutorial be updated because:
- I feel like the tutorial should teach users how to use TFX in a manner that works independently of the type of orchestrator
- it will help users avoid facing unexpected failures when they apply what they learned from the tutorial
- there is currently a lack of clarity on why these artifacts are empty in some situations