Skip to content

Add support for VIDEO as a built-in type#7844

Merged
comfyanonymous merged 9 commits intoComfy-Org:masterfrom
guill:video_input_pr
Apr 29, 2025
Merged

Add support for VIDEO as a built-in type#7844
comfyanonymous merged 9 commits intoComfy-Org:masterfrom
guill:video_input_pr

Conversation

@guill
Copy link
Member

@guill guill commented Apr 28, 2025

This PR adds support for VIDEO as first-class types. In order to avoid unnecessary costs, VIDEO outputs must implement the VideoInput ABC, but their implementation details can vary. Included are two implementations of this type which can be returned by other nodes:

  • VideoFromFile - Created with either a path on disk (as a string) or a io.BytesIO containing the contents of a file in a supported format (like .mp4). This implementation won't actually load the video unless necessary. It will also avoid re-encoding when saving if possible.
  • VideoFromComponents - Created from an image tensor and an optional audio tensor.

Currently, only h264 encoded videos in .mp4 containers are supported for saving, but the plan is to add additional encodings/containers in the near future (particularly .webm).

This PR also adds ImageInput and AudioInput types which can be used for static analysis and type-hinting. Nothing has changed about how these types themselves are represented.

This PR adds support for VIDEO as first-class types. In order to avoid
unnecessary costs, VIDEO outputs must implement the `VideoInput` ABC,
but their implementation details can vary. Included are two
implementations of this type which can be returned by other nodes:

* `VideoFromFile` - Created with either a path on disk (as a string) or
  a `io.BytesIO` containing the contents of a file in a supported format
  (like .mp4). This implementation won't actually load the video unless
  necessary. It will also avoid re-encoding when saving if possible.
* `VideoFromComponents` - Created from an image tensor and an optional
  audio tensor.

Currently, only h264 encoded videos in .mp4 containers are supported for
saving, but the plan is to add additional encodings/containers in the
near future (particularly .webm).
Copy link
Contributor

@huchenlei huchenlei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you attach a workflow which uses all newly added nodes to help testing.

@robinjhuang robinjhuang added Core-Important Core Core team dependency labels Apr 28, 2025
@@ -0,0 +1,318 @@
from __future__ import annotations
from abc import ABC, abstractmethod
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

putting the node types in the comfy folder was a mistake, can you move this out of there and then I'll move the rest.

the comfy folder shouldn't contain anything related to the higher level nodes.

guill added 5 commits April 28, 2025 18:37
I've created a new `comfy_api` top-level module. The intention is that
anything within this folder would be covered by semver-style versioning
that would allow custom nodes to rely on them not introducing breaking
changes.
@guill
Copy link
Member Author

guill commented Apr 29, 2025

video_test_workflow.json

Attached is a workflow that tests all nodes.

Here's an example node that loads a video from a URL to show usage with io.BytesIO:

class LoadVideoFromURL(ComfyNodeABC):
    @classmethod
    def INPUT_TYPES(cls):
        return {
            "required": {
                "url": ("STRING", {"default": "https://mintlify.s3.us-west-1.amazonaws.com/dripart/images/concepts/node/node_appearance.mp4", "tooltip": "The path to the video file."}),
            }
        }
    RETURN_TYPES = (IO.VIDEO,)
    FUNCTION = "load"

    CATEGORY = "image/video"
    DESCRIPTION = "Loads a video from a URL."

    def load(self, url):
        # Download the URL
        import requests
        import io
        response = requests.get(url)
        if response.status_code != 200:
            raise ValueError(f"Failed to download video from {url}. Status code: {response.status_code}")
        # Create a BytesIO object from the response content
        video_data = response.content
        video_io = io.BytesIO(video_data)

        return (VideoFromFile(video_io),)

return None

def filter_files_content_types(files: list[str], content_types: Literal["image", "video", "audio", "model"]) -> list[str]:
def filter_files_content_types(files: list[str], content_types: List[Literal["image", "video", "audio", "model"]]) -> list[str]:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was just fixing an incorrect type annotation.

@guill
Copy link
Member Author

guill commented Apr 29, 2025

I'll also call out this comment on the refactoring commit:

I've created a new comfy_api top-level module. The intention is that
anything within this folder would be covered by semver-style versioning
that would allow custom nodes to rely on them not introducing breaking
changes.

I think it's important that custom nodes feel free to use the VideoInput type and add their own implementations and that we carefully consider any changes that might be backward-incompatible before we make them.

@comfyanonymous comfyanonymous merged commit 68f0d35 into Comfy-Org:master Apr 29, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Core Core team dependency Core-Important

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants