Specification: In-Memory Registry

[Rostifar]

Specification: In-Memory Registry

Design

A MemoryRegistry is a Docker image derived mapping from FeastResource identifiers to in-memory Python objects, making it stateless between pods and deploys. A FeastResource is a union of resource types:

TimeDependentObject = Union[
    BaseFeatureView,
    FeatureView,
    StreamFeatureView,
    OnDemandFeatureView,
    RequestFeatureView,
    Entity,
    FeatureService,
    SavedDataset
]

FeastResource = Union[
    TimeDependentObject, DataSource, SavedDataset, ValidationReference, ProjectMetadata, Infra
]

Some general constraints:

1. MemoryRegistry should subclass BaseRegistry and behave like any other Feast registry during apply calls,
2. When a FeatureStore instance is constructed,MemoryRegistry should be populated by a thread-local feast apply call,
3. apply_* and get_* calls to MemoryRegistry should use / return shallow-copied objects to reduce the risk of side effects,
4. MemoryRegistry instances should be immutable once the is_built flag is set,
5. proto calls to MemoryRegistry should be evaluated on-demand to prevent space overhead,
6. Any MemoryRegistry instance should not exceed 10MB of space usage (as determined by len(dill.dumps(registry_instance)). At the moment (2023/10/10), the registry requires 0.3MB of RAM. MMAPing portions of the registry may be needed as things scale.

[Rostifar]

This is a bug in __copy__. self.source isn't defined for SFV objects; instead, stream_source serves the same role as source.

[Rostifar]

A typing bug: type(self.entities) == List[str] while FeatureView's constructor requires Union[List[Entity], None].