AnimateDiff

AnimateDiff is presented as a framework that proposes an effective framework to animate most of existing personalized text-to-image models once for all. The site states that it is possible for everyone to manifest their imagination into high-quality images with an affordable cost. The approach is to append a newly-initialized motion modeling module to the frozen based text-to-image model, and train it on video clips thereafter to distill a reasonable motion prior.

Gallery pages show best-quality animations generated by models injected with the motion modeling module in our framework, and a supplement demonstrates that our method dose not break the diversity of the original model. The project describes the method as saving the efforts in model-specific tuning.