[Feature]: The Document object needs to support the API for payload

[VinCn80]

There are many scenarios where it is necessary to add custom fields to the vectorized content, such as file name, file creation department, etc. These fields need to be added to the metadata, but the current Document object cannot support this.

[AlbumenJ]

Could you help submit this contribution?

[wuji1428]

I would like to take this, please assign to me.

[wuji1428]

Hello @AlbumenJ ,

While implementing this feature, I observed that documentmeta is a dependency for several upstream components. I would like to seek the guidance on the expected scope of support for these related modules.

Below is my assessment of the potential changes required:

1. Reader Classes

• TextReader
• PDFReader
• WordReader
• ImageReader

Current view: These are not affected by documentmeta changes and, in my opinion, do not require modifications.

2. Store Classes

• QdrantStore
• MilvusStore
• InMemoryStore.

Proposed change: These classes should support the storage and loading of the payload to ensure metadata persistence.

3. Converter Classes

• DifyDocumentConverter
• BailianDocumentConverter
• RAGFlowDocumentConverter
• HayStackDocumentConverter

Problem: These handle third-party data retrieval where search results contain meaningful custom fields. I believe some of these fields are necessary, but I am uncertain whether all of them should be mapped to the payload. For instance, Dify provides a wide range of individual fields.