Single Actor vs Multi-Actor Project Types:
There are four main differences between Single Actor vs Multi-Actor projects
- Audio features
- Ease of Use
- Ability to upload multiple videos to 1 project
- Output video length depends on audio length
Single Actor - video guide
- Single Actor videos. (where only 1 person is visible in the whole video)
Pros | Cons |
Easy to use - Everything is automatic. Simply upload your video and click generate | Lack of control - because everything is automatic, the network may detect a face in the background and it may lip-sync the incorrect person. |
Audio features - automatic translation, text-to-speech, SRT upload. | Can only upload ONE video per project - If you want to lip-sync a 2nd video with the same actor in the same lighting, you must create a new EasyDub project and train a AI model again for that 2nd video. |
Can generate multiple times using the same trained model - this means you will not be charged for training a model for each subsequent result of the same video. | Unlike Multi-Actor projects. Single Actor projects does not have a section to upload additional training footage yet. The training of the AI avatar will only come from the footage that you want to lipsync. So if you upload a 5 second video, then we’ll only use that 5 second video for training an AI model for your result.
This is not ideal for quick small clips since in order to get the best possible results we typically recommend at least 30 seconds of footage that LipDub can train on (where the actor on screen is talking or has some lip shape diversity)
Two workarounds:
1) If you are uploading AI avatar videos —> try adding into your prompt to Kling AI for example “make the actor on screen move their mouths to appear like they are talking naturally with clear inside mouth and teeth texture” (something along those lines). Then loop the video output from Kling AI so a 10 second video becomes like a 1min video!
2) If you are uploading real life actor videos —>Try adding more footage of the actor in the same lighting and same camera angle after the initial small clip that you want to use for lipsync is over, then upload that video to Single Actor Lipdub project. And then be sure that the audio you upload or create for lipsync, only happens during the beginning section where you want the lipsync! |
Match shorter audio files - you have a 1min video and then only upload a 10 second audio that you want to use for lipsync, then the output video will be 10 seconds long. (LipDub assumes you don't need the remaining 50 seconds of silence) |
Multi-Actor - video guide
- Multi Actor videos. (where multiple people’s faces are visible)
Pros | Cons |
More control - users can select each face detection that they want to lip-sync. | No audio features - automatic translation, text-to-speech, and SRT upload are not yet available. User must upload their own dub audio that they want to use for lip-sync. |
Can upload MULTIPLE videos - If you want to lip-sync five videos with the same actor in the same lighting, you can simply upload all videos to one Advanced flow project. | More prone to user error - because it is more in-depth and requires more user input, the chances of user error goes up. |
Can generate multiple times using the same trained model - this means you will not be charged for training a model for each subsequent result of the same video. | Will not match shorter audio files - You have a 1min video and then only upload a 10 second audio that you want to use for lipsync, then the output video will be 1 min long.
Why? - This is common in multi-speaker videos. One person speaks for the first 10 seconds but then is silent the remaining of the video, then another person starts to speak later in the video. So LipDub does not assume that it can cut the remaining 50 seconds, unlike Single-Actor flow |
FAQ - For “Mutli-Actor Projects” can I train a model then upload more videos in the same lighting and same actor and re-use that same model without needing to re-train?
- While this is possible. For the best possible results we recommend you first upload ALL your videos of the actor in the same lighting. Then you only need to train the model once.
- That model will then have seen all the video frames across ALL of your footage, so you when you go to generate, it will get the best possible results!