+ Sravya Popuri☆, Peng-Jen Chen☆, Changhan
+ Wang, Juan Pino, Yossi Adi,
+ Jiatao Gu, Wei-Ning Hsu†, Ann Lee†
+ (☆ = Equal contribution and † = Equal supervision)
+
+
++ We explore self-supervised pre-training with unlabeled speech data and data augmentation to improve direct + Speech to speech model training. We take advantage of a recently proposed speech-to-unit translation (S2UT) + framework that encodes + target + speech into discrete representations, and study both speech encoder and discrete unit decoder pre-training + as well as + efficient partial finetuning methods. We conduct experiments under various data setups and show that + self-supervised + pre-training consistently improves model performance compared with multitask learning and is complementary + to data + augmentation techniques that apply ASR and MT models to create weakly supervised training data. + +
+ + We provide ground truth source and target audios with the corresponding reference text,
+ as well as audio samples from three systems:
+ (1) S2UT+LNA-D: the proposed direct speeech-to-unit translation
+ system initialized with wav2vec 2.0 encoder, unit mBART decoder and finetuned using LNA-D strategy
+ (2) Supervised S2UT: a baseline direct speech-to-unit translation system trained with
+ source and target text as auxiliary task targets.
+
+ (3) S2T+TTS: a baseline cascaded system with a speech-to-text translation model initialized
+ with wav2vec 2.0 encoder and a random decoder, followed by a text-to-speech synthesis model.
+ Both (1) and (2) use an open sourced HiFi-GAN vocoder to convert units to waveforms.
+
| + | Ground truth | +Predictions | +|||
|---|---|---|---|---|---|
| + | Source (Spanish) | +Target (English) | +S2UT+LNA-D | +Supervised S2UT | +S2T+TTS | +
| Sample 1: S2UT+LNAD performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +en dos mil dieciseis hubo tres mil quinientos sesenta y nueve + suicidios en españa según el instituto + nacional de estadística | +IN TWENTY SIXTEEN THERE WERE THREE THOUSAND FIVE HUNDRED SIXTY + NINE SUICIDES IN SPAIN ACCORDING TO + THE NATIONAL INSTITUTE OF STATISTICS | +|||
| ASR: | ++ | + | IN TWENTY SIXTEEN THERE WERE THREE THOUSAND FIVE HUNDRED SIXTY + NINE SUICIDES IN SPAIN ACCORDING TO + THE NATIONAL + INSTITUTE OF STATISTICS | +IN TWENTY SIXTEEN THERE WERE THREE THOUSAND FIVE HUNDRED SIXTY + NINE SIXTY NINE SUICIDES IN SPAIN + ACCORDING TO THE + NATIONAL STATISTICS INSTITUTE | +IN TWENTY SIXTEEN THERE WERE THREE THOUSAND FIVE HUNDRED SIXTY + NINE KILLINGS IN SPAIN ACCORDING TO + THE NATIONAL + STATISTICS INSTITUTE | +
| Sample 2: S2UT+LNAD performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +pero por qué te pusiste en esta situación | +BUT WHY DID YOU PUT YOURSELF IN THIS SITUATION | +|||
| ASR: | ++ | + | BUT WHY DID YOU PUT YOURSELF IN THIS SITUATION | +BUT WHY DID THE SITUATION IN THIS SITUATION | +WHY DID YOU PUT YOU THIS SITUATION | +
| Sample 3: S2UT+LNAD performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +autobuses adicionales normalmente proporcionados por go south coast + van desde bristol al festival + | +ADDITIONAL BUSES USUALLY PROVIDED BY GO SOUTH COAST GO FROM + BRISTOL TO THE FESTIVAL | +|||
| ASR: | ++ | + | ADDITIONAL BUSES USUALLY PROVIDED BY GO SOUTH COAST GO FROM BRISTOL + TO THE FESTIVAL | +ADDITIONAL UP TO BORSES NORMALLY PROVIDED BY COAST SO CAST BANDS OF + BRISTOL ALL FESTIVAL | +ADDITIONAL BUSES USUALLY PROVIDED BY GO SOUTH COAST GO FROM BRUCE + TO THE FESTIVAL | +
| Sample 4: S2UT+LNAD performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +encontró un país con dos gobiernos en la capital maximiliano era el + emperador | +HE FOUND A COUNTRY WITH TWO GOVERNMENTS IN THE CAPITAL MAXIMILIAN + WAS THE EMPEROR | +|||
| ASR: | ++ | + | HE FOUND A COUNTRY WITH TWO GOVERNMENTS IN THE CAPITAL MAXIMILIAN + WAS THE EMPEROR | +HE FOUND A COUNTRY WITH TWO GOVERNMENTS IN THE CAPITAL THE MOST + SIMILIAN CAPITAL WAS THE EMPEROR + | +HE FOUND A COUNTRY WITH TWO GOVERNMENTS AND THE CAPITAL MAXIMILIAN + WAS AN EMPEROR | + +
| Sample 5: S2T+TTS performs best + | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +otro aspecto más institucional es el equilibrio de fuerzas entre + el parlamento y el consejo | +ANOTHER MORE INSTITUTIONAL ASPECT IS THE BALANCE OF POWER BETWEEN + PARLIAMENT AND THE COUNCIL | +|||
| ASR: | ++ | + | ANOTHER MORE INSTITUTIONAL ASPECT IS THE BALANCE OF FORCES BETWEEN + PARLIAMENT AND THE COUNCIL | +ANOTHER MORE INSTITUTIONAL ASPECT IS THE BALANCE OF FORCES BETWEEN + PARLIAMENT AND THE COUNCIL | +ANOTHER MORE INSTITUTIONAL ASPECT IS THE BALANCE OF POWER BETWEEN + PARLIAMENT AND THE COUNCIL | +
| Sample 6: All systems make errors | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +además su capacidad de regeneración es muy limitada | +MOREOVER THEIR CAPACITY FOR REGENERATION IS VERY LIMITED | +|||
| ASR: | ++ | + | MOREOVER ITS CAPACITY FOR REGENERATION IS VERY LIMITED | +IN ADDITION HIS REGENERATION CAPACITY IS VERY LIMITED | +MOREOVER ITS RECOVERY IS VERY LIMITED | +
We provide ground truth source and target audios with the corresponding reference text,
+ as well as audio samples from three systems. All the three models are initialized with wav2vec 2.0 encoder,
+ unit
+ mBART decoder and finetuned using LNA-D strategy but use different datasets for finetuning:
+ (1) S2UT_Base: finetuned on the combination of CoVoST2, Europarl-ST, mTEDx datasets.
+
+ (2) S2UT_LR: finetuned on low resource setup with 50hr of data sampled from the the
+ combination of CoVoST2, Europarl-ST, mTEDx datasets
+
+ (3) S2UT_Aug: finetuned on the the combination of CoVoST2, Europarl-ST, mTEDx datasets
+ datasets plus the ASR data.
+ with wav2vec 2.0 encoder and a random decoder, followed by a text-to-speech synthesis model.
+ All models use an open sourced HiFi-GAN vocoder to convert units to waveforms.
+
| + | Ground truth | +Predictions | +|||
|---|---|---|---|---|---|
| + | Source (Spanish) | +Target (English) | +S2UT_LR | +S2UT_Base | +S2UT_Aug | +
| Sample 1: All systems do well | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +cada uno de ellos es un derecho exclusivo sujeto a ciertas + limitaciones y excepciones + | +EACH ONE OF THEM IS AN EXCLUSIVE RIGHT SUBJECT TO CERTAIN + LIMITATIONS AND EXCEPTIONS | +|||
| ASR: | ++ | + | EACH OF THEM IS AN EXCLUSIVE RIGHT SUBJECT TO CERTAIN LIMITATIONS + AND EXCEPTIONS | +EACH ONE OF THEM IS AN EXCLUSIVE RIGHT SUBJECT TO CERTAIN + LIMITATIONS AND EXCEPTIONS | +EACH OF THEM IS AN EXCLUSIVE RIGHT SUBJECT TO CERTAIN LIMITATIONS + AND EXCEPTIONS | +
| Sample 2: S2UT_LR performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +esta experiencia representa un paso trascendental en la historia + espacial del país | +THIS EXPERIENCE REPRESENTS A TRANSCENDENTAL STEP IN THE SPATIAL + HISTORY OF THE COUNTRY | +|||
| ASR: | ++ | + | THIS EXPERIENCE REPRESENTS A TRANSCENDENT STEP IN THE SPACE HISTORY + OF THE COUNTRY | +THIS EXPERIENCE REPRESENTS A TRANSCENDENTAL STEP IN THE SPATIAL + HISTORY OF THE COUNTRY | +THIS EXPERIENCE REPRESENTS A MOVEMENT STEP IN THE SPACE HISTORY OF + THE COUNTRY | +
| Sample : S2UT_LR has errors but S2UT_Base and S2UT_Aug got + it + right. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +cuando casi diez personas al día una cada dos horas y media | +ALMOST TEN PEOPLE A DAY ONE EVERY TWO AND A HALF HOURS | +|||
| ASR: | ++ | + | ALMOST TEN PEOPLE A DAY ONE EVERY TWO HOURS AND A HALF | +EVEN TEN PEOPLE EVERY DAY ONE EVERY TWO HOURS | +ALMOST TEN PEOPLE A DAY ONE EVERY TWO AND A HALF HOURS | +
| Sample 4: S2UT_Aug performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +desde la perspectiva del balance físico químico y biológico está + en una posición clave | +THE PERSPECTIVE OF PHYSICAL CHEMICAL AND BIOLOGICAL BALANCE IT IS + IN A KEY POSITION | +|||
| ASR: | ++ | + | FROM A PHYSICAL CHEMICAL AND BIOLOGICAL BALANCE HE IS IN A KEY + POSITION | +FROM A PHYSICAL PERSPECTIVE OF PHYSICAL CHEMICAL AND BIOLOGICAL + POSITION | +FROM THE PERSPECTIVE OF PHYSICAL CHEMICAL AND BIOLOGICAL BALANCE IT + IS IN A KEY POSITION | + +
| Sample 5: S2UT_Aug performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| ASR: | ++ | + | AND IT STARTED IN ASIA AND FROM THERE IT PASSED TO VENUS THE MOST + COSMOPOLITAN PORT IN ITS TIME | +HE BECAME IN ASIA AND FROM THAT TIME THE MOST COSMOPOLITAN IN HIS + LIFE | +IT STARTED IN ASIA AND FROM THERE IT WENT TO VENICE THE MOST + COSMOPOLITAN PORT OF ITS TIME | + +
| Reference: | +empezó en asia y de allí pasó a venecia el puerto más cosmopolita + de su tiempo | +IT STARTED IN ASIA AND FROM THERE IT WENT TO VENICE THE MOST + COSMOPOLITAN PORT OF ITS TIME | +|||
| Sample 6: S2UT_Aug performs best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +desde un punto de vista presupuestario no parece adecuada la + propuesta de financiación procedente de la comisión de + desarrollo ya que este dinero no existe al | +IN ANY CASE GIVEN THAT THE FINANCING OF THIS NEW COOPERATION + INSTRUMENT MUST BE COMPATIBLE WITH THE + TWO + THOUSAND SEVEN + TWENTY THIRTEEN FINANCIAL FRAMEWORK IT IS WORTH | +|||
| ASR: | ++ | + | IN ANY CASE GIVEN THAT THE FUNDING OF THIS NEW CORPORATION + INSTRUMENT MUST BE COMPATIBLE WITH THE + TWO THOUSAND SEVEN + TWENTY THIRTEEN FINANCIAL FRAMEWORK IT IS IMPORTANT | +IN ANY CASE SINCE THE FINANCING OF THIS NEW INSTRUMENT OF + CORPORATION MUST COMPATIBLE WITH THE + FINANCIAL FRAMEWORK + FOR TWENTY THIRTEEN | +IN ANY CASE GIVEN THAT THE FINANCING OF THIS NEW CORPORATION + INSTRUMENT MUST BE COMPATIBLE WITH THE + TWO THOUSAND + SEVEN TWENTY THIRTEEN FINANCIAL FRAMEWORK | + +
We provide ground truth source and target audios with the corresponding reference text,
+ as well as audio samples from three systems:
+ (1) S2UT+LNA-D: the proposed direct speeech-to-unit translation
+ system initialized with wav2vec 2.0 encoder, unit mBART decoder and finetuned using LNA-D strategy
+ (2) Supervised S2UT: a baseline direct speech-to-unit translation system trained with
+ source and target text as auxiliary task targets.
+
+ (3) S2T+TTS: a baseline cas2caded system with a speech-to-text translation model
+ initialized
+ with wav2vec 2.0 encoder and a random decoder, followed by a text-to-speech synthesis model.
+ Both (1) and (2) use an open sourced HiFi-GAN vocoder to convert units to waveforms.
+
| + | Ground truth | +Predictions | +|||
|---|---|---|---|---|---|
| + | Source (English) | +Target (Spanish) | +S2UT+LNA-D | +Supervised S2UT | +S2T+TTS | +
| Sample 1: S2UT+LNAD performs the best. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +this should also be an important part of our approach to the twenty + twelve budget | +esto también debería ser una parte importante de nuestro enfoque + del + presupuesto dos mil doce + | +|||
| ASR: | ++ | + | esto también debería ser una parte importante de nuestro enfoque al + presupuesto dos mil doce | +también debería ser una parte importante de nuestro enfoque al + presupuesto dos mil doce | +esto también debería ser una parte importante de nuestro enfoque al + presupuesto de dos mildos mil + dos mil doce | + +
| Sample 2: S2UT+LNAD performs the best. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +information encourages citizens interest in public matters and + their participation | +la información fomenta el interés de los ciudadanos por los + asuntos + públicos y su participación | +|||
| ASR: | ++ | + | la información fomenta el interés de los ciudadanos en asuntos + públicos y su participación | +la información y el interés de los ciudadanos alientan los + intereses de las cuestiones públicas y su + participación + | +la información alienta el interés de los ciudadanos en asuntos + públicos y en su participación | + +
| Sample 3: S2UT+LNAD performs the best. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +and apparently it was quite popular | +y al parecer era muy popular | +|||
| ASR: | ++ | + | y al parecer era muy popular | +y un líder era bastante popular | +y aparentemente era bastante popular | + +
| Sample 4: S2UT+LNAD performs the best. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +his family who are my constituents are convinced of his innocence + | +su familia que son mis electores está convencida de su inocencia + | +|||
| ASR: | ++ | + | su familia que son mis electores está convencida de su inocencia + | +su familia que son mí circunscripciones están convencidas de estos + inocentes | +su familia que son mis electores están convencidos de su inocencia + | +
| Sample 5: All systems do reasonably well. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +we can actually do the same thing with much less energy | +podemos hacer lo mismo con mucha menos energía | +|||
| ASR: | ++ | + | podemos hacer lo mismo con mucha menos energía | +podemos hacer lo mismo que mucha menos energía | +en realidad podemos hacerlo mismo con menos energía | + +
| Sample 6: All systems make errors. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +of the directive on all taxes including social security + contributions the automatic exchange of information and improved + cooperation between the member states in matters of taxation | +de la directiva a todos los impuestos incluidas las + contribuciones a + la seguridad social el intercambio automático de + información y la mejora de la cooperación fiscal entre los estados miembros | +|||
| ASR: | ++ | + | de la directiva a todos los impuestos incluidas las contribuciones + a la seguridad social el + intercambio automático + de información y la mejor cooperación entre los estados miembros en las cuestiones de impuestos | +de la directiva a todos los impuestos impluyendo las contribuciones + de seguridad social el + intercambio automático de + la información y mejorar la cooperación entre los estados miembros y las cuestiones de impuestos + | +de la directiva para todos los impuestos incluidos las + contribuciones de seguridad social el + intercambio automático + de información y la mejor cooperación entre los estados miembros en la cuestión de la fiscalidad + | + +
We provide ground truth source and target audios with the corresponding reference text,
+ as well as audio samples from three systems. All the three models are initialized with wav2vec 2.0 encoder,
+ unit
+ mBART decoder and finetuned using LNA-D strategy but use different datasets for finetuning:
+ (1) S2UT_Base: finetuned on the combination of Europarl-ST, MUST-C datasets.
+
+ (2) S2UT_LR: finetuned on low resource setup with 50hr of data sampled from the combination
+ of Europarl-ST, MUST-C datasets
+
+ (3) S2UT_Aug: finetuned on the combination of Europarl-ST, MUST-C datasets plus the ASR
+ data.
+ with wav2vec 2.0 encoder and a random decoder, followed by a text-to-speech synthesis model.
+ All models use an open sourced HiFi-GAN vocoder to convert units to waveforms.
+
| + | Ground truth | +Predictions | +|||
|---|---|---|---|---|---|
| + | Source (English) | +Target (Spanish) | +S2UT_LR | +S2UT_Base | +S2UT_Aug | +
| Sample Set 1: All systems do well. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +we want to see energy poverty as a part of this debate | +queremos ver la pobreza energética como parte de este debate + | +|||
| ASR: | ++ | + | queremos ver la pobreza energética como parte de este deate | +queremos ver la pobreza energética como parte de este date | +queremos ver la pobreza energética como parte de este deate | +
| Sample Set 2: S2UT_LR has errors but S2UT_Base and S2UT_Aug got + it + right. | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +in my view one of the most important elements is the follow up of + legislative initiative requests from parliament | +en mi opinión uno de los elementos más importantes es el + seguimiento de + las solicitudes de iniciativa legislativa del + parlamento | +|||
| ASR: | ++ | + | n mi opinión uno de los elementos más importantes es el + seguimiento de las peticiones de la + iniciativa legislativa + por parte del pagamento | +en mi opinión uno de los elementos más importantes es el + seguimiento de las emiendas de iniciativas + legislativas de + ley | +en mi opinión uno de los elementos más importantes es el + seguimiento de las solicitudes de + iniciativa legislativa + del pagamento | + +
| Sample Set 3: S2UT_Aug performs the best | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +we must find an open and constructive procedure on the next + financial framework | +debemos encontrar un procedimiento abierto y constructivo en el + próximo marco financiero | +|||
| ASR: | ++ | + | debemos encontrar un procedimiento abierto y constructivo sobre el + próximo marco financiero | +debemos encontrar un procedimiento abierto y constructivo en el + sistema financiero financiero + financiero financiero + | +debemos encontrar un procedimiento abierto y constructivo en el + próximo marco financiero | + +
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +i agree that we should act and react but we should not overdo it + because we need a balanced approach | +estoy de acuerdo en que deberíamos actuar y reaccionar pero no + deberíamos excedernos porque necesitamos un enfoque equilibrado | +|||
| ASR: | ++ | + | estoy de acuerdo en que debemos actuar y reaccionar pero no + debemos hacerlo porque necesitamos un + enfoque + equilibrado | +estoy considerando que actuamos y reaccionamos pero no deberíamos + hacerlo porque necesitamos un + enfoque realmente + valioso | +estoy de acuerdo en que deberíamos actuar y reaccionar pero no + deberíamos exagerarlo porque + necesitamos un enfoque + equilibrado | + +
| Sample Set 4: All systems make errors | +|||||
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +of the directive on all taxes including social security + contributions the automatic exchange of information and improved + cooperation between the member states in matters of taxation | +de la directiva a todos los impuestos incluidas las contribuciones + a la + seguridad social el intercambio automático de + información y la mejora de la cooperación fiscal entre los estados miembros | +|||
| ASR: | ++ | + | de la directiva a todos los impuestos incluidas las contribuciones + a la seguridad social el + intercambio automático + de información y la mejor cooperación entre los estados miembros en las cuestiones de impuestos | +la directiva sobre el impuesto de todos los contribuyentes + inpluyendo las contribuciones sociales la + introducción + automática y mejorada de los estados miembros y mejorar la cooperación entre los estados miembros + | +de la directiva a todos los impuestos incluidas las contribuciones + a la seguridad social el + intercambio automático + de información y la mejor cooperación entre los estados miembros en materia de impuestos | + +
| + | + + + + | ++ + + + | ++ + + + | ++ + + + | ++ + + + | +
| Reference: | +through my work im trying to articulate that humans are not + separate from nature and that everything is interconnected | +a través de mi trabajo estoy tratando de expresar que los humanos + no + están separados de la naturaleza y que todo está + interconectado | +|||
| ASR: | ++ | + | a través de mi trabajo estoy tratando de articular que los humanos + no están separados de la + naturaleza y que todo + está interconectado | +a través de mi trabajo trato de articular que los humanos no somos + separados de la naturaleza y que + todo está + interconectado | +a través de mi trabajo trato de articular que los humanos no + estamos separados de la naturaleza y + que todo está + interconectado | + +