Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

To enable Morpheus to work with Vector databases (VDB), often used with LLM and RAG, a Milvus service and sink should be added. This would allow Morpheus to not only query VDBs in pipelines, but also use Morpheus pipelines to seed VDBs to be used else where.

Milvus has been chosen since it is the most commonly supported VDB between the 3 major LLM libraries: Langchain, llama-index, and Haystack.

Describe your ideal solution

This new feature should include two main components: a service and a sink. The service should function outside of the pipeline and act as a common point to interact with the database. The sink should allow messages to be directly written into the database. The sink will be a user of the service.

Key functionality that is required:

• Milvus Service
  • Capable of configuring access to the database
  • Ability to fetch records from the VDB with optional parameters
  • Ability to write documents to the VDB
  • Capable of reading/writing simultaneously from multiple threads
• Milvus Sink
  • Capable of converting Morpheus messages into the native Milvus format
  • Utilizes the Milvus service to perform the actual writes

Describe any alternatives you have considered

No response

Additional context

Utilize the existing implementations in popular LLM libraries for inspiration:

• Llamaindex: https://gpt-index.readthedocs.io/en/latest/examples/vector_stores/MilvusIndexDemo.html
• Haystack: https://haystack.deepset.ai/integrations/milvus-document-store

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request