Embedding: max token per request error

[srtab]

I'm getting an error when embedding a project. Here's the traceback:

Traceback (most recent call last):
  File "/home/daiv/app/codebase/indexes.py", line 132, in update
    self.semantic_search_engine.add_documents(namespace, documents)
  File "/home/daiv/app/codebase/search_engines/semantic.py", line 98, in add_documents
    document_vectors = self.embeddings.embed_documents([
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/langchain_openai/embeddings/base.py", line 575, in embed_documents
    return self._get_len_safe_embeddings(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/langchain_openai/embeddings/base.py", line 471, in _get_len_safe_embeddings
    response = self.client.create(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 393, in _sentry_patched_create_sync
    return _execute_sync(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 380, in _execute_sync
    raise e from None
  File "/home/daiv/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/openai.py", line 377, in _execute_sync
    result = f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/openai/resources/embeddings.py", line 128, in create
    return self._post(
           ^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1239, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daiv/.venv/lib/python3.12/site-packages/openai/_base_client.py", line 1034, in request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'Requested 810327 tokens, max 300000 tokens per request', 'type': 'max_tokens_per_request', 'param': None, 'code': 'max_tokens_per_request'}}

[srtab]

By debugging the repository causing this error, it was identified an .html file with an base64 image inline. The chunker could not split this document, generating a single chunk with more than 1037299 characters.