feat: Qdrant Search Handler

[anush008]

Description

This PR introduces a new QdrantSearchEnrichmentHandler to add support for vector search using Qdrant.

• Associated unit and integration tests have been added that run locally.

TODO:

• Run ./gradlew :sdks:python:container:generatePythonRequirementsAll - HELP NEEDED, see feat: Qdrant Search Handler #35686 (comment).

[gemini-code-assist]

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[anush008]

Hi @Abacn

Could you please run ./gradlew :sdks:python:container:generatePythonRequirementsAll on this PR branch?
My local machine fails with memory issues and grpcio dependency errors.

[anush008]

Hey @Abacn.
Just bumping this PR. Please take a look when possible.

[derrickaw]

Hi @anush008, can you try cleaning up the lint issues first before @Abacn reviews? Thanks!

[anush008]

Hey @derrickaw
Are those lint failures or issues due to #35686 (comment)?

[derrickaw]

Hey @derrickaw Are those lint failures or issues due to #35686 (comment)?

Good point. I think its a mix of #35686 (comment), but the test ones I believe are lint issues that can be fixed now. Thanks!

[anush008]

I've re-confirmed. Pre-commit on the changes passes fine.

[moawnallah]

My local machine fails with memory issues and grpcio dependency errors.

Hi @anush008, Thanks for putting this Qdrant PR together. It will be helpful in diversifying the RAG integrations currently supported in Beam. Regards this blocking issue, have you tried to run them one by one sequentially? It mostly appear to me the issue is that those generating python docker requirements for different python versions in parallel. That is why it cause memory issues

I remember once in a while I had come across this issue and I solved by this script:

dev@dev:~/beam$ cat ./run_python_requirements.sh
#!/bin/bash

# Array of Python versions to process
versions=(39 310 311 312 313)

# Loop through each version
for version in "${versions[@]}"; do
  log_file="python_${version}_docker_requirements.log"
  echo "Starting sequential Python requirements generation..." >> "${log_file}"
  echo "$(date): Starting process for Python ${version}..." >> "${log_file}"

  # Run the command, redirecting output to the log file.
  ./gradlew :sdks:python:container:py${version}:generatePythonRequirements >> "${log_file}" 2>&1

  # Check if the command was successful
  if [ $? -eq 0 ]; then
    echo "$(date): Completed process for Python ${version} successfully" >> "${log_file}"
  else
    echo "$(date): Process for Python ${version} failed with exit code $?" >> "${log_file}"
  fi
done

echo "$(date): All processes completed" >> "${log_file}"

I am willing to move this PR forward in terms of review work 🙏

[anush008]

Thanks.

[anush008]

Hey @mohamedawnallah
Just bumping this PR. Please review when possible.

[moawnallah]

@anush008 – Currently Docker Image Requirements CI failing. It seems you haven't updated the docker image requirements. Please recheck the comment I posted earlier #35686 (comment) would address that issue

[anush008]

I tried that. Still ran into OOM issues.
I'll have to try with a different machine.

In the meantime, please review when possible.

[moawnallah]

In the meantime, please review when possible.

Usually review started after the author make sure their changes run in the CI workflows without any errors (excluding the flakes) as seeing in a way or another in the contribution guidelines: https://github.com/apache/beam/blob/master/CONTRIBUTING.md

I tried that. Still ran into OOM issues. I'll have to try with a different machine.

It would be helpful for every experiment done to capture logs so we can look at it together here 👍

[moawnallah]

Curious have you tried GitHub Codespace or Gitpod? They come up with a free tier version that is reasonably enough to generate those docker python requirements

[anush008]

Yeah GH Codespace isn't able to make it too. I am looking into the alternative.

[moawnallah]

Yeah GH Codespace isn't able to make it too.

Weird. Just FYI, I am currently on an old Macbook M1 air with 8GB of ram and can generate those python docker requirements

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.