Skip to content

feat(storage): add built-in storage engine with system-level fallback#1468

Open
nixidexiangjiao wants to merge 1 commit into
Tencent:mainfrom
nixidexiangjiao:built-instorage-engine
Open

feat(storage): add built-in storage engine with system-level fallback#1468
nixidexiangjiao wants to merge 1 commit into
Tencent:mainfrom
nixidexiangjiao:built-instorage-engine

Conversation

@nixidexiangjiao
Copy link
Copy Markdown

Description

Add a declarative system-level fallback mechanism for storage engine configuration via config/builtin_storage_engine.yaml, mirroring the existing builtin_models.yaml pattern. When a tenant's storage_engine_config is absent or has an empty provider sub-config, the system automatically falls back to the built-in config.

Key changes:

  • Unified resolver layer (internal/types/storage_resolve.go): ResolveXxxConfig functions implement tenant → builtin → env fallback with empty-shell detection. If a tenant sub-config has all key fields empty (e.g. OSS: {}), it's treated as an empty shell and bypassed in favor of the builtin fallback.
  • YAML loader (internal/types/builtin_storage_engine_config.go): Loads the config file at startup with ${ENV} and ${ENV:-default} placeholder interpolation, stored in an atomic.Pointer singleton.
  • Env interpolation (internal/types/env_interpolation.go): ${VAR} for string fields (unset stays literal to surface misconfig), ${VAR:-default} for bool/int fields (unset falls back to default to avoid YAML parse errors).
  • Factory & handler refactor: Replace scattered sec == nil || sec.Xxx == nil checks with types.ResolveXxxConfig(sec) calls in factory.go and system.go.
  • Frontend fix: KBStorageSettings.vue now checks props.storageProvider instead of local state for provider pre-selection.
  • OBS deprecation: The legacy OBS_* env-var path is now the 3rd-tier fallback (tenant → builtin → env) and emits a sync.Once deprecation warning. Planned for removal in vip-v0.7.x.
  • YAML struct tags: All *EngineConfig types now have yaml tags alongside existing json tags, enabling deserialization from both API/DB (JSON) and config file (YAML).

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 💥 Breaking change
  • 📚 Documentation update
  • 🎨 Refactor
  • ⚡ Performance improvement
  • 🧪 Test
  • 🔧 Configuration / Build / CI

Related Issue

Fixes #

Testing

All new test files pass with go test ./internal/types/ ./internal/application/service/file/ ./internal/handler/:

  • internal/types/env_interpolation_test.go${VAR} and ${VAR:-default} expansion, unset/empty/default cases
  • internal/types/storage_resolve_test.go — Full matrix: tenant-first, empty-shell fallback, partial-config-no-fallback, both-nil; plus OBS env deprecation and sync.Once once-only guarantee
  • internal/types/builtin_storage_engine_config_test.go — File missing, invalid YAML, env interpolation, empty config, env override path, atomic replace
  • internal/application/service/file/factory_builtin_test.go — All 8 providers with builtin fallback, default provider fallback, both-nil returns incomplete
  • internal/handler/system_storage_resolve_test.goisXxxConfigured with builtin fallback for all providers
  • internal/handler/tenant_storage_resolve_test.goGetTenantStorageEngineConfig fills DefaultProvider from builtin, respects STORAGE_ALLOW_LIST, never leaks builtin secrets to client

Checklist

  • make fmt && make lint && make test pass locally
  • Self-reviewed the code
  • Added/updated tests covering the change
  • Updated related documentation (README, docs/, Swagger annotations, etc.)
  • Breaking changes are clearly called out in the description above

Screenshots / Recordings

No user-visible UI changes (only a minor fix to provider pre-selection logic in the KB storage settings).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant