Repository navigation
dimension mismatch error on weight loading #4
Description
Activity
The issue is because
./vocab/SpeAss_IO_label.vocabhas two labels "O" and "ARG2", while checkpoints have higher (32774) output vocabulary size.It may be because of different versions of the python library. I have updated the requirements.txt. Could you please re-install your library again? Thanks.
Now it hits an unkown "idna" package
https://github.com/ncbi/GNorm2/blob/main/requirements.txt#L34I tried with the updated requirements. It breaks due to stanza==1.6.1 (downgrading to 1.4.0 helped).
After that, I am still getting the same dimension mismatch error.
What is the number of predicted tokens expected of the model in
/src_python/SpeAss/model_sa.py:HUGFACE_NER? Is it 2 ("O" and "ARG2") or 32774?Here is the listing of layers in the final model in
/src_python/SpeAss/model_sa.py. Seemingly it tries to load something that belongs into the last layer in the very first layer. Or it confuses two first weights.One possibility: it may stem from
_legacy_weightsin the actual keras version (2.15.0), but it is what's in the requirements.txtlayer: tf_bert_model_1/bert/embeddings/word_embeddings/weight:0 (32774, 512) layer: tf_bert_model_1/bert/embeddings/token_type_embeddings/embeddings:0 (2, 512) layer: tf_bert_model_1/bert/embeddings/position_embeddings/embeddings:0 (512, 512) layer: tf_bert_model_1/bert/embeddings/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/embeddings/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._0/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._0/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._0/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._0/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._1/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._1/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._1/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._1/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._2/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._2/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._2/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._2/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._3/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._3/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._3/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._3/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._4/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._4/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._4/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._4/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._5/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._5/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._5/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._5/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._6/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._6/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._6/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._6/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/self/query/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/self/query/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/self/key/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/self/key/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/self/value/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/self/value/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/output/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/attention/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/intermediate/dense/kernel:0 (512, 2048) layer: tf_bert_model_1/bert/encoder/layer_._7/intermediate/dense/bias:0 (2048,) layer: tf_bert_model_1/bert/encoder/layer_._7/output/dense/kernel:0 (2048, 512) layer: tf_bert_model_1/bert/encoder/layer_._7/output/dense/bias:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/output/LayerNorm/gamma:0 (512,) layer: tf_bert_model_1/bert/encoder/layer_._7/output/LayerNorm/beta:0 (512,) layer: tf_bert_model_1/bert/pooler/dense/kernel:0 (512, 512) layer: tf_bert_model_1/bert/pooler/dense/bias:0 (512,) layer: softmax/kernel:0 (512, 2) layer: softmax/bias:0 (2,)I confirm this is the same error I experienced - issue #2. I managed to fix it by using
python 3.8, settingprotobuf>=3.19.0,<=3.20.xinrequirements.txt, and therefore installingtensorflow==2.3.0, but I guess that didn't help in your case :/I am getting this error while trying to run
GNorm2.sh:Traceback (most recent call last): File "/home/dlituiev/GNorm2/GeneNER_SpeAss_run.py", line 716, in <module> speciesAss(args.NERoutpath,args.SAoutpath, args.SAmodel) File "/home/dlituiev/GNorm2/GeneNER_SpeAss_run.py", line 623, in speciesAss nn_model.load_model(modelfile) File "/home/dlituiev/GNorm2/src_python/SpeAss/model_sa.py", line 103, in load_model self.model.load_weights(model_file) File "/home/dlituiev/GNorm2/gnorm2/lib64/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/home/dlituiev/GNorm2/gnorm2/lib64/python3.9/site-packages/keras/src/backend.py", line 4359, in _assign_value_to_variable variable.assign(value) ValueError: Cannot assign value to variable ' tf_bert_model_1/bert/embeddings/word_embeddings/weight:0': Shape mismatch.The variable shape (32774, 512), and the assigned value shape (2, 512) are incompatible. Total 0 file(s) wait(s) for process.This may be related to issue #2, but it's hard to tell as they provide no error trace
Did you ever figure it out? I'm having the same issue. I downgraded stanza to deal with the package error and now I'm stuck on the shape mismatch error.
@ChihHsuanWei would you be able to share a Dockerfile or a docker image for reproducibility?
Sorry for late response. I am asking the main author of the species assignment module to involve in. I will update you soon.
Reacted by Dmytro Lituiev, jessicapatricoski and Fauzi MaulanaHi, GNorm2 somehow has a difficultly within Python 3.9. we prepared the requirments.txt for python 3.8 and 3.10. Could you please reinstall the appropriate lib? Let me know if any further issue?
requirements-py38.txt
requirements-py310.txtThanks. Wei
Is there a configuration for 3.11? My computing cluster only has 3.9 and 3.11
Update: I was able to overcome that obstacle by using a conda environment specifying 3.10 and it appears to be working! Thank you!
The conda environment workaround prohibits GPU use
I am getting this error while trying to run
GNorm2.sh:This may be related to issue #2, but it's hard to tell as they provide no error trace