Speaker-and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech

Weise T, Klumpp P, Demir K, Perez Toro PA, Schuster M, Nöth E, Heismann B, Maier A, Yang SH (2024)

Publication Language: English

Publication Type: Conference contribution

Publication year: 2024

Publisher: International Speech Communication Association

Series: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Pages Range: 1545-1549

Conference Proceedings Title: Interspeech 2024

Event location: Kos Island GR

DOI: 10.21437/Interspeech.2024-1208

Abstract

This paper introduces a novel combination of two tasks, previously treated separately: acoustic-to-articulatory speech inversion (AAI) and phoneme-to-articulatory (PTA) motion estimation.We refer to this joint task as acoustic phoneme-to-articulatory speech inversion (APTAI) and explore two different approaches, both working speaker-and text-independently during inference.We use a multi-task learning setup, with the end-to-end goal of taking raw speech as input and estimating the corresponding articulatory movements, phoneme sequence, and phoneme alignment.While both proposed approaches share these same requirements, they differ in their way of achieving phoneme-related predictions: one is based on frame classification, the other on a two-staged training procedure and forced alignment.We reach competitive performance of 0.73 mean correlation for the AAI task and achieve up to approximately 87 % frame overlap compared to a state-of-the-art text-dependent phoneme force aligner.

Authors with CRIS profile

Tobias Weise Professur für Artificial Intelligence in Biomedical Speech Processing (Stiftungsprofessur) Philipp Klumpp Professur für Informatik (Mustererkennung) Kubilay Demir Lehrstuhl für Informatik 5 (Mustererkennung) Paula Andrea Perez Toro Lehrstuhl für Informatik 5 (Mustererkennung) Elmar Nöth Professur für Informatik (Mustererkennung) Björn Heismann Lehrstuhl für Informatik 5 (Mustererkennung) Andreas Maier Lehrstuhl für Informatik 5 (Mustererkennung) Seung Hee Yang Professur für Artificial Intelligence in Biomedical Speech Processing (Stiftungsprofessur)

Involved external institutions

Ludwig-Maximilians-Universität (LMU) DE Germany (DE)

How to cite

APA:

Weise, T., Klumpp, P., Demir, K., Perez Toro, P.A., Schuster, M., Nöth, E.,... Yang, S.H. (2024). Speaker-and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech. In Interspeech 2024 (pp. 1545-1549). Kos Island, GR: International Speech Communication Association.

MLA:

Weise, Tobias, et al. "Speaker-and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech." Proceedings of the 25th Interspeech Conferece 2024, Kos Island International Speech Communication Association, 2024. 1545-1549.

BibTeX: Download