Discuss on Discord (keeping it on linktree now so it's always an active link)
This version improves video init. You can now generate optical flow maps from input videos, and use those to:
- warp init frames for consistent style
- warp processed frames for less noise in final video
The feature works like this: we take the 1st frame, diffuse it as usual as an image input with fixed skip steps. Then we warp in with its flow map into the 2nd frame and blend it with the original raw video 2nd frame. This way we get the style from heavily stylized 1st frame (warped accordingly) and content from 2nd frame (to reduce warping artifacts and prevent overexposure)
- Add existing flow check, now generate only if no flow found
- Add comprehensive error reporting for missing video_init, video frames, flow files
- Fix non alphanumeric batch names not working
- Fix frames not being sorted before creating output video
- Fix incorrect RAFT root foder on local machines
- Add storing RAFT on gdrive
- Add colab for 3d animation only
- Add saving frames and flow to google drive (suggested by Chris the Wizard#8082)
- Add back consistency checking
- Update 512x512 and secondary model urls
- Remove consistency checking for stability
- Add 256x256 comis faces model
- Add ViT-L/14@336px
- Add warp parameteres to saved settings
- Use width_height size instead of input video size
- Bring back adabins and 2d/3d anim modes
- Install RAFT only when video input animation mode is selected
- Generate optical flow maps only for video input animation mode even with flow_warp unchecked, so you can still save an obtical flow blended video later
- Install AdaBins for 3d mode only (should do the same for midas)
- Add animation mode check to create video tab
(Located in animation settings tab)
Video Optical Flow Settings:
- flow_warp - check to warp
- flow_blend: 0 - you get raw input, 1 - you get warped diffused previous frame
- check_consistency: check forward-backward flow consistency (uncheck unless getting too many warping artifacts)
This feature is plain simple - we just take any frame, warp in to the next frame, blend with real next frame, get smooth noise-free result.
(located in create video tab) blend_mode:
- none: just mash frames together in a video
- optical flow: take frame, warp, blend with the next frame
- check_consistency: use consistency maps (may prevent warping artfacts)
- blend: 0 - you get raw 2nd frame, 1 - you get warped 1st frame
- Add automatic flow map management (i.e. create only when needed)
- Add error reporting for missing inputs, flows, frames
- Add turbo
- Add turbosmooth
This is a variation of the awesome DiscoDiffusion colab
If you like what I'm doing you can check my linktree
Thank you for being awesome!
