feat(rag): add ExternalApiReader for third-party document parsing int…

[magicyuan876]

Closes #317

This PR adds a generic ExternalApiReader to enable seamless integration with third-party document parsing services in the RAG module.

Changes Made

• Add generic external API document parsing reader
• Support custom HTTP request construction and response parsing
• Support polling for asynchronous task completion
• Include comprehensive tests and MinerU Tianshu integration example
• Support multiple document formats (PDF, DOCX, PPTX, XLSX, TXT)

AgentScope-Java Version

1.0.4-SNAPSHOT

Description

Background:
The existing RAG module lacks a flexible way to integrate with external document parsing services. This limitation prevents users from leveraging powerful third-party APIs like MinerU Tianshu, cloud OCR services, or enterprise document processing systems.

Solution:
This PR introduces ExternalApiReader, a highly configurable reader that uses functional interfaces to adapt to any external API. It extends AbstractChunkingReader to maintain consistency with existing readers while providing maximum flexibility.

Key Features:

• Functional Interfaces: RequestBuilder and ResponseParser allow users to customize HTTP requests and response parsing logic
• Async Task Support: Built-in polling mechanism for APIs that use async task patterns
• Retry Mechanism: Configurable retry with exponential backoff for reliability
• Format Validation: Validates file formats before processing
• Comprehensive Error Handling: Custom exceptions with detailed error messages

Example Usage:
ExternalApiReader reader = ExternalApiReader.builder()
.requestBuilder((filePath, client) -> {
// Custom request building
return new Request.Builder()
.url("https://api.example.com/parse")
.header("Authorization", "Bearer TOKEN")
.post(createRequestBody(filePath))
.build();
})
.responseParser((response, client) -> {
// Custom response parsing
return extractMarkdown(response);
})
.chunkSize(512)
.splitStrategy(SplitStrategy.PARAGRAPH)
.build();

List docs = reader.read(ReaderInput.fromString("file.pdf")).block();Testing:

• Added 434 lines of comprehensive unit tests
• Included real-world integration example with MinerU Tianshu (363 lines)
• All tests pass with proper mocking of HTTP interactions

Checklist

Please check the following items before code is ready to be reviewed.

• Code has been formatted with mvn spotless:apply
• All tests are passing (mvn test)
• Javadoc comments are complete and follow project conventions
• Related documentation has been updated (e.g, links, examples, etc.)
• Code is ready for review

Additional Notes

Files Added:

• ExternalApiReader.java (444 lines) - Core implementation
• ExternalApiReaderTest.java (434 lines) - Comprehensive unit tests
• MinerUTianshuReaderExample.java (363 lines) - Production-ready integration example

Integration Example:
The included MinerUTianshuReaderExample demonstrates integration with MinerU Tianshu (https://github.com/magicyuan876/mineru-tianshu), a powerful open-source document parsing service. This example can serve as a template for integrating other similar services.

Benefits:

• Enables RAG applications to leverage state-of-the-art document parsing services
• Provides a reusable pattern for future API integrations
• Maintains consistency with AgentScope-Java's reactive programming model
• No breaking changes to existing code

Related Issue: #317

[cla-assistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

This PR introduces a generic ExternalApiReader to enable seamless integration with third-party document parsing services in the RAG module. The implementation follows a functional interface approach with customizable HTTP request building and response parsing.

Key Changes:

• Adds ExternalApiReader with functional interfaces (RequestBuilder and ResponseParser) for flexible API integration
• Implements retry mechanism with exponential backoff and configurable timeouts
• Includes comprehensive unit tests with MockWebServer for HTTP interactions
• Provides a complete MinerU Tianshu integration example demonstrating async task polling

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 19 comments.

File	Description
ExternalApiReader.java	Core implementation extending AbstractChunkingReader with builder pattern, retry logic, and support for custom HTTP client configuration
ExternalApiReaderTest.java	Comprehensive unit tests covering sync/async APIs, authentication, multipart uploads, retry behavior, and custom interceptors
MinerUTianshuReaderExample.java	Production-ready integration example with MinerU Tianshu API showing task submission, polling, and various configuration options

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[AlbumenJ]

Please fix test

[codecov]

Codecov Report

❌ Patch coverage is 70.47619% with 31 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../agentscope/core/rag/reader/ExternalApiReader.java	70.47%	27 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

[magicyuan876]

Please fix test

done