Variational Test-time Optimization for Diffusion Synchronization

[Project Website] [arXiv]

Hyunsoo Lee*, Farrin Marouf Sofian*, Kushagra Pandey, Stephan Mandt

(* Equal contribution)

Introduction

SyncVC synchronizes diffusion trajectories at test time by optimizing controls, enabling coherent wide images, optical illusions, and 3D mesh textures without training.

Installation

First, clone this repository:

git clone https://github.com/mandt-lab/SyncVC
cd SyncVC

Then, create a conda environment:

conda create -n syncvc python=3.10 -y
conda activate syncvc

Install PyTorch 2.4.1+cu118 and PyTorch3D:

python -m pip install -U uv
uv pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu118
uv pip install iopath fvcore ninja "setuptools<80"
FORCE_CUDA=1 uv pip install -v --no-build-isolation "git+https://github.com/facebookresearch/pytorch3d.git@stable"

Install the remaining dependencies:

uv pip install -r requirements.txt

Running the Code

Wide Image Generation

python wide_image_generation.py \
    --prompt "A photo of a mountain range at twilight." \
    --random_seed 0 --num_images 1 --use_float_16 --save_dir "./results/wide_image" 

You can generate different wide images by modifying the --prompt argument.

Optical Illusion Generation

python illusion_generation.py \
    --prompt_1 "A photo of a horse" \
    --prompt_2 "A photo of a snowy mountain village" \
    --random_seed 0 --num_images 1 --use_float_16  
    --save_dir "./results/illusion" --illusion_transform_type "ccw" 

You can try different semantic pairs by modifying --prompt_1 and --prompt_2, and different optical illusion transforms by modifying --illusion_transform_type.

Text-guided 3D Mesh Texturing

python mesh_texturing.py \
    --prompt "a jeep" --save_dir ./results/mesh --tag jeep \
    --mesh "./data/jeep.obj" --random_seed 0

You can texture your own mesh by modifying the --mesh and --prompt arguments.

Troubleshooting an error related to CuPy

If you encounter an error related to CuPy, try the following command:

export LD_LIBRARY_PATH="$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cuda_nvrtc/lib:$CONDA_PREFIX/lib/python3.10/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH"

Acknowledgements

This repository is based on the following works:

• diffusers
• SyncTweedies
• SyncSDE
• Visual Anagrams
• StyleID
• Objaverse-XL

We thank the authors for making their repositories publicly available.

Citation (BibTeX)

@article{lee2026variational,
    title={Variational Test-time Optimization for Diffusion Synchronization},
    author={Lee, Hyunsoo and Sofian, Farrin Marouf and Pandey, Kushagra and Mandt, Stephan},
    journal={arXiv:2606.15614},
    year={2026}
}