'SentenceTransformer' object has no attribute 'get_input_embeddings'

[Z4HRA-S]

sentence-transformer == 3.1.1
torch == 2.4.1+cu121
transformers == 4.44.2
I get this error when I run trainer.train() on colab.
I have a custom_model as described in this doc.

The model I have used:
meta-llama/Meta-Llama-3.1-8B-Instruct
and
intfloat/e5-large

The full error:

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

[<ipython-input-13-596e2e7dce1c>](https://localhost:8080/#) in <cell line: 2>()
----> 2 trainer.train()

2 frames

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in __getattr__(self, name)
   1727             if name in modules:
   1728                 return modules[name]
-> 1729         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1730 
   1731     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'SentenceTransformer' object has no attribute 'get_input_embeddings'

[tomaarsen]

Hello!

I think the error is missing some traceback frames. Could you perhaps share those?
SentenceTransformers indeed doesn't have any get_input_embeddings attribute anywhere, although that does exist in transformers: https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20get_input_embeddings&type=code

• Tom Aarsen

[Z4HRA-S]

@tomaarsen This is the error trace that trainer.train() returned. I was using some training arguments that SentenceTransformers seems to not support!
Here is my earlier training args:

output_dir=".",
    learning_rate=lr,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,

    eval_accumulation_steps=100, 

    report_to="tensorboard",
    num_train_epochs=1,
    weight_decay=wd,
    eval_strategy="steps",
    eval_steps=80,
    logging_strategy="steps",
    logging_steps=10,
    load_best_model_at_end=True,
    gradient_accumulation_steps=gr_acc,
    remove_unused_columns=False,
    fp16= fp_16_enabled,
    neftune_noise_alpha=5.0,
    warmup_steps = warmup_steps,
    save_steps=80,
    save_only_model=True

And here is the arguments that actually works:

output_dir=".",

    num_train_epochs=1,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,

    gradient_accumulation_steps=accumulation_step,
    eval_accumulation_steps=100, #This one solves OOM during evaluation
    learning_rate=3e-4,
    warmup_ratio=0.1,
    weight_decay=0.01,

    fp16=True,
    bf16=False,

    eval_strategy="steps",
    eval_steps=eval_step,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=2,
    logging_steps=log_step,
    report_to="tensorboard",

    load_best_model_at_end=True,