Structured data learning with TabTransformer Tutorial Working on Colab, broken on Local #1946
Assignees
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue Type
Bug
Source
source
Keras Version
Keras 3.4.1
Custom Code
No
OS Platform and Distribution
macOS Ventura 13.6.3
Python version
3.10.12
GPU model and memory
Apple M2 Pro
Current Behavior?
After running the cell with
run_experiment()I expect the model to train and continue with the tutorial. Instead it errors out (log below).I would expect running this locally vs colab to have the same working result. I have tried to ensure that my local has the same Python version, Keras version, numpy version, etc as the Colab notebook to reduce the amount of possible culprits for the error.
Based on some debugging, I think that the model is not correctly mapping the named inputs to their named input layers. ie: it appears that an input like
native_countryis being passed to therelationshipinput (45 is not in [0, 6)).Standalone code to reproduce the issue or tutorial link
Relevant log output
{ "name": "InvalidArgumentError", "message": "Graph execution error: Detected at node functional_7_1/embedding_12_1/GatherV2 defined at (most recent call last): File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 196, in _run_module_as_main File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 86, in _run_code File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in <module> File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 739, in start File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/tornado/platform/asyncio.py\", line 205, in start File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/events.py\", line 80, in _run File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 545, in dispatch_queue File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 534, in process_one File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 362, in execute_request File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 778, in execute_request File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 449, in do_execute File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3075, in run_cell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3130, in _run_cell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3334, in run_cell_async File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3517, in run_ast_nodes File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3577, in run_code File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/232252535.py\", line 1, in <module> File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/1288716379.py\", line 24, in run_experiment File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 318, in fit File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 121, in one_step_on_iterator File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 108, in one_step_on_data File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 51, in train_step File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 175, in call File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/function.py\", line 171, in _run_through_graph File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 556, in call File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/core/embedding.py\", line 140, in call File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/numpy.py\", line 4875, in take File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py\", line 1951, in take indices[0] = 45 is not in [0, 6) \t [[{{node functional_7_1/embedding_12_1/GatherV2}}]] [Op:__inference_one_step_on_iterator_13726]", "stack": "--------------------------------------------------------------------------- InvalidArgumentError Traceback (most recent call last) Cell In[15], line 1 ----> 1 history = run_experiment( 2 model=baseline_model, 3 train_data_file=train_data_file, 4 test_data_file=test_data_file, 5 num_epochs=NUM_EPOCHS, 6 learning_rate=LEARNING_RATE, 7 weight_decay=WEIGHT_DECAY, 8 batch_size=BATCH_SIZE, 9 ) Cell In[8], line 24, in run_experiment(model, train_data_file, test_data_file, num_epochs, learning_rate, weight_decay, batch_size) 21 validation_dataset = get_dataset_from_csv(test_data_file, batch_size) 23 print(\"Start training the model...\") ---> 24 history = model.fit( 25 train_dataset, epochs=num_epochs, validation_data=validation_dataset 26 ) 27 print(\"Model training finished\") 29 _, accuracy = model.evaluate(validation_dataset, verbose=0) File ~/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs) 119 filtered_tb = _process_traceback_frames(e.__traceback__) 120 # To get the full stack trace, call: 121 # `keras.config.disable_traceback_filtering()` --> 122 raise e.with_traceback(filtered_tb) from None 123 finally: 124 del filtered_tb File ~/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 51 try: 52 ctx.ensure_initialized() ---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 54 inputs, attrs, num_outputs) 55 except core._NotOkStatusException as e: 56 if name is not None: InvalidArgumentError: Graph execution error: Detected at node functional_7_1/embedding_12_1/GatherV2 defined at (most recent call last): File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 196, in _run_module_as_main File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/runpy.py\", line 86, in _run_code File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel_launcher.py\", line 18, in <module> File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/traitlets/config/application.py\", line 1075, in launch_instance File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelapp.py\", line 739, in start File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/tornado/platform/asyncio.py\", line 205, in start File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 603, in run_forever File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/base_events.py\", line 1909, in _run_once File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/asyncio/events.py\", line 80, in _run File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 545, in dispatch_queue File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 534, in process_one File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 362, in execute_request File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/kernelbase.py\", line 778, in execute_request File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/ipkernel.py\", line 449, in do_execute File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3075, in run_cell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3130, in _run_cell File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3334, in run_cell_async File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3517, in run_ast_nodes File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py\", line 3577, in run_code File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/232252535.py\", line 1, in <module> File \"/var/folders/pt/yxcmy21n099708ptn0358ntw0000gn/T/ipykernel_20599/1288716379.py\", line 24, in run_experiment File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 318, in fit File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 121, in one_step_on_iterator File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 108, in one_step_on_data File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py\", line 51, in train_step File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 175, in call File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/function.py\", line 171, in _run_through_graph File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/models/functional.py\", line 556, in call File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/layer.py\", line 882, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 117, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/operation.py\", line 46, in __call__ File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py\", line 156, in error_handler File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/layers/core/embedding.py\", line 140, in call File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/ops/numpy.py\", line 4875, in take File \"/Users/blakebennett/anaconda3/envs/tabtransformer-venv/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py\", line 1951, in take indices[0] = 45 is not in [0, 6) \t [[{{node functional_7_1/embedding_12_1/GatherV2}}]] [Op:__inference_one_step_on_iterator_13726]" }The text was updated successfully, but these errors were encountered: