Comments/Suggestions

[astoniab]

Thanks for this project. I was looking for a way to run Kokoro via onnxruntime. Here are some comments and suggestions about what I've experienced.

1. When passing a filename to load the onnx model, the filename (char_t *) gets cast to a (wchar_t *). That crashes, as least on Windows. You'd need to convert the multibyte string to wide char string.
2. I'd like to be able to use different onnxruntime execution providers to run the model. I've hacked this into mine by passing a new parameter for execution provider and its parameters when creating the session. I also had to re-export the phonemizer and Kokoro models since each had an issue using the DirectML execution provider.
3. It would be great to also support the KittenTTS model. It's almost a drop in replacement. Just a few differences with the input size and vocab.
4. I re-exported the vocab from Kokoro, and got a slightly different one than in this repository. I didn't notice any output differences, so I'm not sure if it matters.
5. Some words with apostrophes (e.g. doesn't, won't) aren't pronounced correctly. I don't see them in the dictionary, so I don't know if it's an issue with the input to the phonemizer model, or the output of it.

[danemadsen]

Hi, thanks for the input.

Issue 1 seems pretty easy to fix, i can do that for you.

I'm not sure what you mean by "onnxruntime execution provider".

I can support kittentts if you like, ive worked with it in another project, its a pretty good model.

The models included in this repo do differ to the normal export in that their output includes phoneme timings so i can lipsync to the output audio.

The mispronounced words is because they arnt in the dictionary, im trying to find all the words that are mispronounced and put the correct pronunciations in the dict. Its a huge undertaking though.

[astoniab]

Hi, thanks for the input.

Issue 1 seems pretty easy to fix, i can do that for you.

I'm not sure what you mean by "onnxruntime execution provider".

Execution Providers

They determine which hardware runs the model. So if you run the model on an execution provider that supports GPUs, it would execute faster than on the CPU. I've been using DirectML for various models, and it executes on any GPU that supports DX 12 for Windows. It's only a few extra lines of code to setup the execution provider, but they do need some parameters to configure depending on which execution providers you're using.

I can support kittentts if you like, ive worked with it in another project, its a pretty good model.

The models included in this repo do differ to the normal export in that their output includes phoneme timings so i can lipsync to the output audio.

The mispronounced words is because they arnt in the dictionary, im trying to find all the words that are mispronounced and put the correct pronunciations in the dict. Its a huge undertaking though.

Are you using espeak to generate the phonemes, or something else? When I tried on the huggingface kokoro example space, it pronounces correctly, so that's why I was wondering why the openphonemizer model doesn't get it correct.

[danemadsen]

Are you using espeak to generate the phonemes, or something else?

No, thats what open phonemizer is for. eSpeak has a much larger dictionary and does a better job at G2P but i cant use it in this project because its GPL licensed which is incompatible with this projects MIT license. The GPL is a copyleft license so any projects that use a GPL projects code must also be GPL, this makes code from GPL projects largely unusable for people who want to deploy software commercially. If you wanted to hook eSpeak up and use it for G2P you can and it may work better but it's not something ill be doing in this project due to the license incompatibility.

One of the goals of this project is to produce something with a permissive MIT license that can be used as an alternative to eSpeak.

[astoniab]

Are you using espeak to generate the phonemes, or something else?

No, thats what open phonemizer is for. eSpeak has a much larger dictionary and does a better job at G2P but i cant use it in this project because its GPL licensed which is incompatible with this projects MIT license. The GPL is a copyleft license so any projects that use a GPL projects code must also be GPL, this makes code from GPL projects largely unusable for people who want to deploy software commercially. If you wanted to hook eSpeak up and use it for G2P you can and it may work better but it's not something ill be doing in this project due to the license incompatibility.

One of the goals of this project is to produce something with a permissive MIT license that can be used as an alternative to eSpeak.

I wasn't asking about what you're using in this repository to generate phonemes. I'm asking what you're using when you come across a word that isn't pronounced correctly and you add it to the dictionary. Obviously the openphonemizer model didn't generate the correct phonemes so something else is needed to generate them.

[danemadsen]

I wasn't asking about what you're using in this repository to generate phonemes. I'm asking what you're using when you come across a word that isn't pronounced correctly and you add it to the dictionary.

Ohhh, haha. I'm using ChatGPT or similar for that. Or sometimes i do it manually to make sure it sounds right.