Here’s a few common issues, and ways to improve them.
- Articulation issues (e.g. incorrect lip shape for certain words)
- Translation with voice clone doesn't sound like a native speaker.
- Audio alignment is incorrect
- Missing lip-sync (e.g. a small portion of the video is not lip-synced!)
- Missing lip-sync (e.g. my entire video is not lip-synced)
- Color differences
- Texture sliding
- Pixilation in the mask edge
If your issue is not addressed here, or if you are still confused on how to improve the result, please don’t hesitate to reach out support@lipdub.ai or message us on our Discord Server.
1. Articulation issues
LipDub AI is an audio driven tool. I.e. the lip shapes that are pasted back on the original video is based on what audio that you give the model.
Example: If you provide different audio takes of the exact same line but different emphasis on certain sounds like the “B” sound in “Birthday”, the articulation will be different!
Example 2: Pitching down the audio by 20% might create for better articulation as well! 🙂
Be sure that there is no music or SFX in the background of the dub audio. This will interfere with the articulation.
2. Translation with voice clone doesn’t sound like a native speaker
This is a possibility when voice cloning.
- Example: If I voice clone an native Spanish speaker and translate them into English. Then the result may sound like a English speaker with a bit of a Spanish accent.
- While this sounds more natural to the original voice, it does not sound like a native speaker.
If you’d like to sound like an native speaker, then please let us know at support@lipdub.ai. And we’ll prioritize adding a feature in the “Translate with AI” option where you can select from a dropdown of voices similar to our current Text-to-Speech functionality.
3. Audio alignment is incorrect
This is a possibility with translation with AI. The exact same sentence in Spanish may be 20% longer than in English.
- Example: If I were to translate an English speaker to Spanish, the audio of the Spanish may not be perfectly aligned with the same hand movements as in the original English video.
The long term solution is to have an timeline editor directly the platform where you can move the newly created audio so it aligns with the video correctly before you generate. This is on our roadmap. Cannot provide an ETA at this time.
Short term solution is to use an editor like Adobe Premiere or DaVinci to align the audio before uploading it to the LipDub platform.
4. Missing lip-sync (small portion of the video)
What is happening? - This is common issue when using the Advanced Flow project. If it is happening in the Advanced flow project, this issue caused by face detections not being properly labelled.
Solution - Double check all the face detections for your actor, and that they are grouped correctly. Be sure to double check the unassigned detections.
5. Missing lip-sync (entire video)
What is happening? - the face in the background is actually being lip-synced, and not the main actress. This is a known face detection issue that can sometimes happen with EasyDub. LipDub face detection is highly sensitive, so its always best to ensure only 1 face is viewable to prevent this from happening.
Solution:
- Please restart this project using Advanced Workflow Guide. This advanced workflow gives users ability to select face detection tracks of which face you'd like to lip-sync, and which ones you do not want to lip-sync.
- Or If you really want to use EasyDub you can! But please be sure to upload either a different video where only 1 face is visible, or with a matte box over the background actors.
6. Texture Sliding / Grain issues
What is happening? - The grain in the original footage may be interfering when LipDub tries to paste back on the mask area.
Solution - If you can denoise the footage, re-upload and re-train the model on just de-noised data, it may improve the result!
7. Color Difference
What is happening? - This can happen if the color space is not supported by LipDub. Rec. 709 and sRGB are the color spaces that supported by LipDub AI.
View all specifications below:
Solution - Please ensure your video color space and your formats are supported by LipDub AI.
If you are experiencing color differences that is not related to a color space issues, please reach out to support@lipdub.ai or message us on our discord server.
8. Pixilation on the mask edge
What is happening? - Sometimes the mask edge can be pixilated due to the face size being too small. LipDub crops into the face of the actor in the video and then performs the lip-sync. If the size of the face is already so small in frame than this pixilation can occur since there is very few pixels to create for a smooth transition in the mask edge.
Solution - Upload a larger video resolution.
If you are experiencing pixilation in the mask edge that is not related to a small face, please reach out to support@lipdub.ai or message us on our discord server.