Skip to content

refactor: Refactor S3 configuration structure and fix Python 3.9 compatibility issues#73

Merged
MaojiaSheng merged 2 commits intovolcengine:mainfrom
baojun-zhang:s3_format
Feb 5, 2026
Merged

refactor: Refactor S3 configuration structure and fix Python 3.9 compatibility issues#73
MaojiaSheng merged 2 commits intovolcengine:mainfrom
baojun-zhang:s3_format

Conversation

@baojun-zhang
Copy link
Collaborator

@baojun-zhang baojun-zhang commented Feb 5, 2026

Description

Refactor S3 configuration structure and fix Python 3.9 compatibility issues

Related Issue

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • Extract S3-related configurations from AGFSConfig into a new S3Config class
  • Update agfs_manager.py to use the new S3Config structure
  • Modify test_config_validation.py to adapt to the new configuration structure
  • Update configuration example files to reflect the new configuration structure
  • Fix Python 3.9 compatibility issue in embedding_queue.py by replacing EmbeddingMsg | None with Optional[EmbeddingMsg]

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

image

Additional Notes

ov.conf example

{
  # .....
  "storage": {
    "agfs": {
      "port": 8080,
      "log_level": "warn",
      "path": "./data",
      "backend": "s3",
      "url": "http://localhost:8080",
      "timeout": 10,
      "retry_times": 3,
      "s3": {
        "bucket": "openviking-test",
        "endpoint": "http://127.0.0.1:8333",
        "region": "us-east-1",
        "access_key": "any",
        "secret_key": "any",
        "prefix": "",
        "use_ssl": "false"
      }
    }
  }
}

…atibility issues

- Extract S3-related configurations from AGFSConfig into a new S3Config class
- Update agfs_manager.py to use the new S3Config structure
- Modify test_config_validation.py to adapt to the new configuration structure
- Update configuration example files to reflect the new configuration structure
- Fix Python 3.9 compatibility issue in embedding_queue.py by replacing `EmbeddingMsg | None` with `Optional[EmbeddingMsg]`
@MaojiaSheng MaojiaSheng merged commit 3ffde58 into volcengine:main Feb 5, 2026
2 of 3 checks passed
MaojiaSheng pushed a commit that referenced this pull request Feb 6, 2026
* feat: 增加对私部vikingdb的支持

* feat: add search_with_sparse_logit_alpha (#71)

* refactor: Refactor S3 configuration structure and fix Python 3.9 compatibility issues (#73)

* refactor: Refactor S3 configuration structure and fix Python 3.9 compatibility issues
- Extract S3-related configurations from AGFSConfig into a new S3Config class
- Update agfs_manager.py to use the new S3Config structure
- Modify test_config_validation.py to adapt to the new configuration structure
- Update configuration example files to reflect the new configuration structure
- Fix Python 3.9 compatibility issue in embedding_queue.py by replacing `EmbeddingMsg | None` with `Optional[EmbeddingMsg]`

* fix: validate_config validate error

* fix: fix ci (#74)

* refactor: unify async execution utilities into run_async (#75)

- Create centralized run_async() in openviking/utils/async_utils.py
- Remove duplicate _run_async from session.py
- Remove duplicate run_coroutine_sync from observers/async_utils.py
- Replace asyncio.run() calls in sync_client.py with run_async
- Update observers to use the unified run_async utility
- Update tests and documentation for new observer API

* 原生部署的vikingdb由外部来管理

---------

Co-authored-by: kkkwjx <zhoujiahui.01@bytedance.com>
Co-authored-by: baojun-zhang <zhangbaojun.1@bytedance.com>
Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants