Refine the text cleaning before embedding the documents in the RAG pipeline

[yonitoo]

Overview

Our current text cleaning method converts the text to lower case, removes punctuation, lemmatizes and removes the stop words from the text. As discussed HERE, the transformer models (in our case SentenceTransformer) doesn't require such extensive preprocessing, it's even suggested to not do it as this way some context might be lost.

Suggested solution
Drop the lemmatization and stop words removal from the cleaning.
Double-check if the lower case conversion isn't done by default by the transformer model we are using.
The cleaning step is something you would expect to have in a pipeline, so we need to figure out how to handle it properly.
Decide on what text cleaning logic might be relevant and add it.

Acceptance criteria

Remove the extensive NLP preprocessing (lemmatization and stop words removal).
Add relevant text cleaning logic.