Cross-lingual Transfer of Phonological Features for Low-resource Speech Synthesis

Dan Wells, Korin Richmond
The Centre for Speech Technology Research
University of Edinburgh

This is the companion webpage for our paper on using phonological feature vectors as inputs for text-to-speech synthesis.

Our paper was accepted to the 2021 ISCA Speech Synthesis Workshop! You can read it here.

Listening test samples

The table below presents a selection of the audio samples used in our listening tests. Each row represents a single test panel from our MUSHRA experiment, where participants rated each sample on a naturalness scale from 0–100.

System key

• Ref – Natural speech reference
• Copy – Vocoded speech using mel spectral features extracted from the reference (copy synthesis)
• P – Phoneme inputs
• F – Phonological feature inputs
• 240 – Trained on 4 hours of German speech only
• 240-ft – Base model 14 hours of English, fine-tuned using 4 hours of German
• 15-ft – Base model 14 hours of English, fine-tuned using 15 minutes of German

Overall, our best performing TTS systems were P-240-ft and F-240-ft, with no significant difference between the two. There was also no significant difference between fine-tuned systems using just 15 minutes of German data and those trained from scratch on 4 hours of German. While this suggests that fine-tuning a source model from a high-resource language is a viable approach in low-resource settings, we find no evidence that one input representation is better than the other in this context when it comes to subjective naturalness ratings. However, we did find that phonological features give a more natural voice when training on 4 hours of German speech only (F-240 > P-240). For more details on model setup and interpreting the results, please see the paper.

Input characters representing phonemes unique to German and unseen during English model training are highlighted below in red.

Transfer from English to German

The table below presents the progression of our P-240-ft and F-240-ft German voices as the corresponding English base models are fine-tuned. The first column represents zero-shot synthesis of German speech from the English base model. When using phoneme inputs, embeddings for German-specific phonemes are initialised with the embeddings of similar English phonemes from the base model. With phonological feature inputs, representations of German-specific phonemes are composed from representations of individual features in the base model directly.

Note: We re-initialise the stop token predictor of our Tacotron 2 model at the start of fine-tuning, so early checkpoints are more prone to utterance-final babbling than the source English model synthesising German phone strings on its own.