This research covers the following topics:
-
Explore a custom LoRA implementation of BERT attention layers for accelerated model training, with a focus on MLM task, and train an SVR model on the obtained embeddings to solve a regression task.
-
Evaluate LSTM model performance in comparison to the custom LoRA implementation.
-
Highlight the impact of additional feature engineering on model performance.
-
Present a performance comparison to underscore the effectiveness of the custom LoRA implementation in enhancing model outcomes.