Add support for VIDEO as a built-in type#7844
Add support for VIDEO as a built-in type#7844comfyanonymous merged 9 commits intoComfy-Org:masterfrom
Conversation
This PR adds support for VIDEO as first-class types. In order to avoid unnecessary costs, VIDEO outputs must implement the `VideoInput` ABC, but their implementation details can vary. Included are two implementations of this type which can be returned by other nodes: * `VideoFromFile` - Created with either a path on disk (as a string) or a `io.BytesIO` containing the contents of a file in a supported format (like .mp4). This implementation won't actually load the video unless necessary. It will also avoid re-encoding when saving if possible. * `VideoFromComponents` - Created from an image tensor and an optional audio tensor. Currently, only h264 encoded videos in .mp4 containers are supported for saving, but the plan is to add additional encodings/containers in the near future (particularly .webm).
huchenlei
left a comment
There was a problem hiding this comment.
Can you attach a workflow which uses all newly added nodes to help testing.
comfy/comfy_types/input_types.py
Outdated
| @@ -0,0 +1,318 @@ | |||
| from __future__ import annotations | |||
| from abc import ABC, abstractmethod | |||
There was a problem hiding this comment.
putting the node types in the comfy folder was a mistake, can you move this out of there and then I'll move the rest.
the comfy folder shouldn't contain anything related to the higher level nodes.
I've created a new `comfy_api` top-level module. The intention is that anything within this folder would be covered by semver-style versioning that would allow custom nodes to rely on them not introducing breaking changes.
|
Attached is a workflow that tests all nodes. Here's an example node that loads a video from a URL to show usage with class LoadVideoFromURL(ComfyNodeABC):
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"url": ("STRING", {"default": "https://mintlify.s3.us-west-1.amazonaws.com/dripart/images/concepts/node/node_appearance.mp4", "tooltip": "The path to the video file."}),
}
}
RETURN_TYPES = (IO.VIDEO,)
FUNCTION = "load"
CATEGORY = "image/video"
DESCRIPTION = "Loads a video from a URL."
def load(self, url):
# Download the URL
import requests
import io
response = requests.get(url)
if response.status_code != 200:
raise ValueError(f"Failed to download video from {url}. Status code: {response.status_code}")
# Create a BytesIO object from the response content
video_data = response.content
video_io = io.BytesIO(video_data)
return (VideoFromFile(video_io),) |
| return None | ||
|
|
||
| def filter_files_content_types(files: list[str], content_types: Literal["image", "video", "audio", "model"]) -> list[str]: | ||
| def filter_files_content_types(files: list[str], content_types: List[Literal["image", "video", "audio", "model"]]) -> list[str]: |
There was a problem hiding this comment.
This was just fixing an incorrect type annotation.
|
I'll also call out this comment on the refactoring commit:
I think it's important that custom nodes feel free to use the |
This PR adds support for VIDEO as first-class types. In order to avoid unnecessary costs, VIDEO outputs must implement the
VideoInputABC, but their implementation details can vary. Included are two implementations of this type which can be returned by other nodes:VideoFromFile- Created with either a path on disk (as a string) or aio.BytesIOcontaining the contents of a file in a supported format (like .mp4). This implementation won't actually load the video unless necessary. It will also avoid re-encoding when saving if possible.VideoFromComponents- Created from an image tensor and an optional audio tensor.Currently, only h264 encoded videos in .mp4 containers are supported for saving, but the plan is to add additional encodings/containers in the near future (particularly .webm).
This PR also adds
ImageInputandAudioInputtypes which can be used for static analysis and type-hinting. Nothing has changed about how these types themselves are represented.