first-order-model VS Wav2Lip

First Order Motion Model/Thin Plate Spline (Animate Single images realistically using a driving video) https://github.com/AliaksandrSiarohin/first-order-model (FOMM - Animate still images using driving videos) https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model (Thin Plate Spline - Likely just a repost of FOMM but with better documentation and tutorials on YouTube) https://drive.google.com/drive/folders/1PyQJmkdCsAkOYwUyaj_l-l0as-iLDgeH (FOMM/Thin Plate Checkpoints) https://disk.yandex.com/d/lEw8uRm140L_eQ (FOMM/Thin Plate Checkpoints mirror) -------3D ANIMATION--------

GitHub - AliaksandrSiarohin/first-order-model: This repository contains the source code for the paper First Order Motion Model for Image Animation

Not sure about whatever bot op is specifically using, but you can use this software to achieve the same result https://github.com/AliaksandrSiarohin/first-order-model

I think the novel and interesting tech is still happening, its just that without the colorful ads for it on TV, and without the software being packaged up and sold with pretty box art that you can physically hold, it doesn't feel as much like a capital-E Experience. It's probably the Internet's fault that we don't do things like that anymore, but the upside is that we now have access to so many ideas and applications from all over, even ones that aren't commercially viable.

Some that look exciting to me are: an AI that lets you animate still photos realistically [1], a simple website that guides you to discover new parks, eateries, and other places near you [2], an AI that colorizes old black-and-white photos/video [3], a Street View style map of the game world from "The Legend of Zelda: Breath of the Wild", with some 1st person 360 degree photos [4], and a tiny game engine that lets you distribute your whole game physically via printed QR codes [5].

If marketing and graphic design people ever felt like getting together to do some 'side projects', I vote that they should make print ads for apps/websites that they like :)

[1] https://github.com/AliaksandrSiarohin/first-order-model

[2] https://randomlocation.xyz (https://randomlocation.xyz/help.txt for customization)

[3] https://github.com/jantic/DeOldify

[4] https://nassimsoftware.github.io/zeldabotwstreetview/

[5] https://github.com/kesiev/rewtro

Ai Used https://github.com/AliaksandrSiarohin/first-order-model

First Order Model

First Order Motion Model (Easy): You can give it an input image and a source video, and the movements from the video will be copied to the image. It has some janky-ness and I would recommend using an ai upscaller afterwards because the resolution isn't that high https://github.com/AliaksandrSiarohin/first-order-model

Hey HN, we’re sync. (https://synclabs.so/). We’re building fast + lightweight audio-visual models to create, modify, and understand humans in video.

You can check our more about us and our company in this video here: https://bit.ly/3TV27rd

Our first api lets you lip-sync a person in a video to an audio in any language in zero-shot. You can check out some examples here (https://bit.ly/3IT3UXk)

Here’s a demo showing how it works and how to sync your first video / audio: https://bit.ly/4ablRwo

Our playground + api is live, you can play with our models here: https://app.synclabs.so/

Four years ago we open-sourced Wav2lip (https://github.com/Rudrabha/Wav2Lip), the first model to lipsync anyone to any audio w/o having to train for each speaker. Even now, it’s the most prolific lipsyncing model to date (almost 9k GitHub stars).

Human lip-sync enables interesting features for many products – you can use it to seamlessly translate videos from one language to another, create personalized ads / video messages to send to your customers, or clone yourself so you never have to record a piece of content again.

We’re excited about this area of research / the models we’re building because they can be impactful in many ways:

[1] we can dissolve language as a barrier

check out how we used it to dub the entire 2-hour Tucker Carlson interview with Putin speaking fluent English: https://vimeo.com/914605299

imagine millions gaining access to knowledge, entertainment, and connection — regardless of their native tongue.

realtime at the edge takes us further — live multilingual broadcasts + video calls, even walking around Tokyo w/ a Vision Pro 2 speaking English while everyone else Japanese.

[2] we can move the human-computer interface beyond text-based-chat

keyboard / mice are lossy + low bandwidth. human communication is rich and goes beyond just the words we say. what if we could compute w/ a face-to-face interaction?

Many people get carried away w/ the fact LLMs can generate, but forget they can also read. The same is true for these audio/visual models — generation unlocks a portion of the use-cases, but understanding humans from video unlocks huge potential.

Embedding context around expressions + body language in inputs / outputs would help us interact w/ computers in a more human way.

[3] and more

powerful models small enough to run at the edge could unlock a lot:

eg.

If your technically inclined you can use https://github.com/Rudrabha/Wav2Lip to sync the lip movements to the new audio.

This is the Github link : https://github.com/Rudrabha/Wav2Lip

Video mapping onto Audio: Now you have Audio with coherent back and forth dialogue. To get the looped video puppets, you find a relatively stable interview clip (in this channel and many of Athenes other ones, the clips of the people just stay in one place). Then feed the audio + video clip into a lipsync algorithm like this https://bhaasha.iiit.ac.in/lipsync/

It's not animating really, just lip sync and face restoration, here I used: https://github.com/Rudrabha/Wav2Lip and https://github.com/TencentARC/GFPGAN respectively.

I just used the web based wav2lip demo. https://bhaasha.iiit.ac.in/lipsync/ Haven’t used the plan in a while, however the colab gives much better results. This was just a quick dusty example done all in the phone.

https://github.com/Rudrabha/Wav2Lip Demo: https://youtu.be/0fXaDCZNOJc