Skip to content

Schema datasets #242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jun 12, 2024
Merged

Schema datasets #242

merged 16 commits into from
Jun 12, 2024

Conversation

Oxid15
Copy link
Owner

@Oxid15 Oxid15 commented May 15, 2024

This one changes how data validation is handled in cascade. It introduces schemas in datasets.

Now pipeline blocks can explicitly define input schemas as pydantic models (optional dependency for this feature). If schema is defined in new SchemaModifier then it will implicitly insert ValidationWrapper before itself in the pipeline. When using self._dataset inside its own __getitem__ it will wrap itself in the validator which will perform validation of the input.
For example we have a dataset of annotated images:

class AnnotImage(pydantic.BaseModel):
    image: List[List[List[float]]]
    segments: List[List[int]]
    bboxes: List[Tuple[int, int, int, int]]

Base class for the pipeline will be:

class ImagesDataset(SchemaModifier):
    in_schema = AnnotImage

Let's define a dummy modifier:

class IDoNothing(ImagesDataset):
    def __getitem__(self, idx):
        item = self._dataset[idx]
        return item

Let's say we have a source of images and
hope that it will maintain our schema. The following
is the regular way we use modifier, but under the hood
it will automatically check that the output of the
datasource is AnnotImage

ds = SomeImageDatasource()
ds = IDoNothing(ds)

@Oxid15 Oxid15 self-assigned this May 15, 2024
@Oxid15 Oxid15 merged commit 1e73047 into 0.14.0 Jun 12, 2024
@Oxid15 Oxid15 deleted the schema_datasets branch June 12, 2024 08:58
@Oxid15 Oxid15 mentioned this pull request Aug 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant