Skip to content

Fix speech editing boundary artifacts by working in mel domain - #1242

Merged
SWivid merged 1 commit into
SWivid:mainfrom
whk-sjtu:fix-speech-edit-mel-domain
Dec 26, 2025
Merged

Fix speech editing boundary artifacts by working in mel domain#1242
SWivid merged 1 commit into
SWivid:mainfrom
whk-sjtu:fix-speech-edit-mel-domain

Conversation

@whk-sjtu

Copy link
Copy Markdown

Previously, speech_edit.py worked in wav domain (inserting zeros into the waveform before computing mel spectrogram), which caused boundary artifacts when mel spectrogram windows straddled zeros and real audio.

This commit refactors the approach to work in mel domain:

  • Compute mel spectrogram on the clean original audio first
  • Insert zero frames in mel domain instead of zero samples in wav domain
  • Use frame-level granularity throughout for consistency

Benefits:

  • Eliminates boundary artifacts
  • More consistent behavior regardless of small float variations in input times
  • Cleaner edit boundaries

Changes to speech_edit.py (lines 148-220):

  • Convert audio to mel using model.mel_spec() before editing
  • Build mel_cond by concatenating original mel frames + zero frames
  • Calculate all time-based values at frame level first, then convert to samples
  • Pass mel_cond directly to model.sample() instead of raw audio

Previously, speech_edit.py worked in wav domain (inserting zeros into the waveform before computing mel spectrogram), which caused boundary artifacts when mel spectrogram windows straddled zeros and real audio.

This commit refactors the approach to work in mel domain:
- Compute mel spectrogram on the clean original audio first
- Insert zero frames in mel domain instead of zero samples in wav domain
- Use frame-level granularity throughout for consistency

Benefits:
- Eliminates boundary artifacts
- More consistent behavior regardless of small float variations in input times
- Cleaner edit boundaries

Changes to speech_edit.py (lines 148-220):
- Convert audio to mel using model.mel_spec() before editing
- Build mel_cond by concatenating original mel frames + zero frames
- Calculate all time-based values at frame level first, then convert to samples
- Pass mel_cond directly to model.sample() instead of raw audio
@SWivid
SWivid merged commit 27e20fc into SWivid:main Dec 26, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants