Skip to content

Refactor Registry & RegistryBackend API #307

@JeremyWurbs

Description

@JeremyWurbs

Description

Refactor the RegistryBackend API to do the following:

  1. Support batch operations;
  2. Move all locking operations to be internal to each backend;
  3. Remove overwrite as part of the public API;

Explicitly, the RegistryBackend API should be:

# Objects API
push(self, name: str | list[str], version: str | list[str], local_path: str | Path | list[str | Path], metadata: dict | list[dict] | None = None)
pull(self, name: str | list[str], version: str | list[str], local_path: str | Path | list[str | Path])
delete(self, name: str | list[str], version: str | list[str])

# Object Metadata API
save_metadata(self, name: str | list[str], version: str | list[str], metadata: dict | list[dict])
fetch_metadata(self, name: str | list[str], version: str | list[str]) -> dict[str, dict]
delete_metadata(self, name: str | list[str], version: str | list[str])

# Serialization
register_materializer(self, object_class: str | list[str], materializer_class: str | list[str])
registered_materializers(self, object_class: str | None | list[str]) -> dict[str, str]

# Registry-level Metadata
save_registry_metadata(self, metadata: dict)
fetch_registry_metadata(self) -> dict

# Discovery
list_objects(self) -> list[str]
list_versions(self, name: str | list[str]) -> dict[str, list[str]]
has_object(self, name: str | list[str], version: str | list[str]) -> dict[str, bool]

# Utility Methods
validate_object_name(self, name: str | list[str]) -> None  # raise exception if any names fail

The associated Registry methods should also support batch operations:

save(self,
    name: str | list[str],
    obj: Any | list[Any],
    *,
    materializer: Type[BaseMaterializer] | None | list[Type[BaseMaterializer]] | list[None] = None,
    version: str | list[str] | None = None,
    init_params: Dict[str, Any] | None | list[Dict[str, Any] | None] = None,
    metadata: Dict[str, Any] | None | list[Dict[str, Any] | None] = None,
)

# similarly for load, delete, info, etc.

Acceptance Criteria

  • Complete Registry implementation
  • Complete support for the three main backends: Local, Minio and GCP
  • 100% unit test coverage
  • Expanded integration test suite

Priority

High (blocking release or critical path)

Estimated Effort (Story Points)

8 (1 week or more)

Metadata

Metadata

Assignees

Labels

mindtrace-registryIssues raised from registry module in mindtrace package

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions