Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement keyframe transitions #120

Open
jbilcke-hf opened this issue Sep 17, 2024 · 0 comments
Open

Implement keyframe transitions #120

jbilcke-hf opened this issue Sep 17, 2024 · 0 comments
Assignees
Labels
mvp you da most valuable playa new feature
Milestone

Comments

@jbilcke-hf
Copy link
Owner

Priority

This is a feature that is all the rage now, so I don't think we should wait 2 months to implement this.

Generally, I would say that new AI/features should be implemented in < 2 weeks time spans (à la Hugging Face and Replicate) so that we do not become too old (AI trends move fast)

Context

Some AI models support keyframe transitions (there are a lot of ComfyUI workflows to do this, with animatediff & co)

Some do it particularly well, for instance Luma Labs: https://x.com/LumaLabsAI/status/1835742817060331783

The good news that that Clapper isn't too far from being able to do this: this can be done by implementing a ClapSegmentCategory.TRANSITION resolver, that would take the shots on both ends (start and end) and gracefully merge them using the selected transition workflow (ClapWorkflowCategory.VIDEO_TRANSITION)

Solution

  • add a rule when resolving a transition to grab the shots at both ends
  • use the keyframing system from LumaLabs to implement it
@jbilcke-hf jbilcke-hf added new feature mvp you da most valuable playa labels Sep 17, 2024
@jbilcke-hf jbilcke-hf self-assigned this Sep 17, 2024
@jbilcke-hf jbilcke-hf added this to the 1.0.0 (MVP) milestone Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mvp you da most valuable playa new feature
Projects
Status: No status
Development

No branches or pull requests

1 participant