Looks like it uses a Japanese voicepack to generate English lyrics

[lethanner]

Continuing the topic #15.

When I don't install Japanese voices from https://github.com/numediart/MBROLA-voices, it crashes with failed to load voice "ja".
But the lyrics I provided is https://github.com/ASLP-lab/DiffRhythm/blob/main/infer/example/eg_en.lrc which are entirely in English.

So, probably because it uses Japanese voicepack, the generated lyrics practically does not correspond to the provided lyrics - only fragments of words are recognizable.

Here is an example of generated audio (packed to zip, wav converted to 192k mp3 using ffmpeg)
output.zip

[Jandown]

I have also encountered the same problem. Do you have any other better solutions?

[lethanner]

I FOUND THE REAL REASON why it need to install the Japanese package!!

Launched step-by-step debug and found out:

1. It's correctly determines a lyrics language code.
2. Exception raises while all hardcoded phonemizers beginning initialization.
3. This is only true for Japanese, as it is the only language that uses the mbrola library (among languages coded specifically in DiffRhythm). All other languages ​​in the initialization keeps using the modules embedded into eSpeak.

Take a look at the list of languages in the DiffRhythm code:

lang2backend = {
 "zh": phonemizer_zh, # using "cmn" code
 "ja": phonemizer_ja,
 "en": phonemizer_en, # using "en-US" code
 "fr": phonemizer_fr, # using "fr-fr" code
 "ko": phonemizer_ko,
 "de": phonemizer_de,
}

Now look at the return of EspeakWrapper.available_voices() (truncated):

{
 'cmn': 'sit\\cmn',
 'ja': 'mb\\mb-jp1', # pay attention to this only mb
 'en-us': 'gmw\\en-US',
 'fr-fr': 'roa\\fr',
 'ko': 'ko',
 'de': 'gmw\\de'
}

And this code from phonemizer library:

if 'mb' in voice_code: # <-- did you notice?
 # this is an mbrola voice code. Select the voice by using
 # identifier in the format 'mb/{voice_code}'
 available = {
 voice.identifier[3:]: voice.identifier
 for voice in self.available_voices('mbrola')}
else:
 # this are espeak voices. Select the voice using it's attached
 # language code. Consider only the first voice of a given code as
 # they are sorted by relevancy
 available = {}
 for voice in self.available_voices():
 if voice.language not in available:
 available[voice.language] = voice.identifier
# (truncated)
if self._espeak.set_voice_by_name(voice_name.encode('utf8')) != 0:
 raise RuntimeError( # pragma: nocover
 f'failed to load voice "{voice_code}"')

Do you understand what I'm getting at?

So, this model most likely DOES NOT using Japanese voicepack for English lyrics.
And the problem of illegible lyrics lies somewhere else.
(perhaps this topic should even be renamed)

Will look further.

[lethanner]

FINALLY ABOUT ILLEGIBLE LYRICS!
(...why the issue closed so early? :))

I compared official ASLP-lab inference code with https://github.com/newgenai79/DiffRhythm-windows and my last guess was confirmed.
This is an encoding issue.

In newgenai79's repository there are many encoding='utf-8' added to Python's "open" methods. Looks like Python on Linux loads files in UTF-8 by default, but not on Windows.

After manually appending encoding='utf-8' to opens, I got much more correctly English speaking. Yes, there are still problems with some words, but in any case, the most important thing is that the words now exist at all.
I'm ready to open a pull request.

You can also listen proof in the archive:
output.zip

Demo lyrics are generated by DeepSeek.

[00:00.00] Downloaded DiffRhythm, ready to go,
[00:04.00] But eSpeak alone? It’s a no-show.
[00:08.00] MBrola’s needed, a library to find,
[00:12.00] Installed it quick, but peace? Not in my mind.
[00:16.00] Crashed on launch, searching Japanese,
[00:20.00] But lyrics are English, why the tease?
[00:24.00] Installed the pack, still errors galore,
[00:28.00] "No such file or dir," what’s it looking for?
[00:32.00] Process Monitor, my debugging friend,
[00:36.00] Found missing folders, brought them in the end.
[00:40.00] Copied from Linux, eSpeak and MBrola too,
[00:44.00] Launched DiffRhythm, but the vocals were stew.
[00:48.00] Thought it was Japanese, loading wrong,
[00:52.00] Debugged the code, the truth came along.
[00:56.00] Hardcoded future, Japanese in the way,
[01:00.00] Devs removed it, but still no pay.
[01:04.00] File encoding, the final fight,
[01:08.00] UTF-8 forced, and it worked just right.
[01:12.00] Now DiffRhythm sings, clear as can be,
[01:16.00] AI music on Windows, finally free.