feat: Support input/output files by introducing FileReference Pydantic annotation

[linusseelinger]

Relevant issue or PR

N/A

Description of changes

We'd like to support (binary) input/output files in tesseracts. This PR adds a FileReference Pydantic annotation.

Its validation step ensures the file exists and is a path relative to correct directory. The latter is crucial since it ensures that files are actually accessible, AND ensures we won't leak internals of the container/host.

Usage (based on hello world tesseract):

class InputSchema(BaseModel):
    name: str = Field(description="Name of the person you want to greet.")
    input_file: FileReference = Field(description="A file that can be used as input.")

[...]

def apply(inputs: InputSchema) -> OutputSchema:
    with inputs.input_file.open() as f:
        file_content = f.read()
        print(f"File content: {file_content}")
    [...]

curl -d '{"inputs": {"name": "Osborne", "input_file": "../helloworld/test.txt"}}' -H "Content-Type: application/json" http://127.0.0.1:8080/apply

The tesseract will resolve this to simply "test.txt" and verify the file exists.

TODO

• Remove example code in helloworld
• add dedicated example + docs for input/output files
• Automatically move output files to (upcoming) --output-path, and ensure paths are relative to --input-path and --output-path. Depends on @angela-ko 's upcoming changes.

Testing done

Local testing for variations of file paths.

[codecov]

Codecov Report

Attention: Patch coverage is 42.10526% with 11 lines in your changes missing coverage. Please review.

Project coverage is 76.31%. Comparing base (15511c8) to head (00c5d80).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
tesseract_core/runtime/schema_types.py	42.10%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #240      +/-   ##
==========================================
- Coverage   76.52%   76.31%   -0.21%     
==========================================
  Files          28       28              
  Lines        3156     3175      +19     
  Branches      505      508       +3     
==========================================
+ Hits         2415     2423       +8     
- Misses        520      531      +11     
  Partials      221      221              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nmheim]

Just reading this before our meeting and leaving my thoughts here so they do not get lost (sorry for commenting on a draft). Maybe we can do this with a field validator?

from typing import Annotated
from pydantic import AfterValidator, RootModel

def is_relative(path: Path) -> Path:
        if path.is_relative_to(Path.cwd()):
            path = path.relative_to(Path.cwd())
        else:
            raise ValueError(
                f"FileReference path must be relative to the current working directory: {path}"
            )
    return path


class FileReference(RootModel):
    root: Annotated[Path, AfterValidator(is_relative)]

[linusseelinger]

I played around with RootModel a bit. The key issue I run into is that for a schema like

class InputSchema(BaseModel):
    name: str = Field(description="Name of the person you want to greet.")
    input_file: FileReference = Field(description="A file that can be used as input.")

the server gets the validated model

name='Osborne' input_file=Apply_FileReference(root=PosixPath('test.txt'))

I don't see a straightforward way to get rid of the nested model and resolve straight to a path. In order to avoid nesting, my understanding is that we actually have to go with the custom type (which also has the upside that we can even pass plain strings and automatically convert them to Path objects).

[nmheim]

hmm, does this help?

In [4]: m = InputSchema(name="Osborne", input_file="./test.txt")

In [5]: m.model_dump()
Out[5]: {'name': 'Osborne', 'input_file': PosixPath('test.txt')}

In [6]: m.input_file.model_dump()
Out[6]: PosixPath('test.txt')

[linusseelinger]

Indeed, that gets rid of the nested model! The catch is that in apply, the user gets an InputSchema (not a JSON/dict), so we can't dump the input before passing to user code (and I don't think the user should have to unpack inputs).

I've pushed the RootModel experiment to https://github.com/pasteurlabs/tesseract-core/tree/linus/feat-file-schema-root-model

You can give it a try by serving the helloworld example and

curl -d '{"inputs": {"name": "Osborne", "input_file": "test.txt"}}' -H "Conn/json" http://127.0.0.1:8080/apply