CMI slides Feb05

Information about CMI slides Feb05

Published on November 1, 2007

Author: Lucianna

Source: authorstream.com

Content

Multilingual Conversational Systems:  Multilingual Conversational Systems Steps to Develop Language Learning System:  Steps to Develop Language Learning System Begin with existing mature system in English Develop English-to-Mandarin translation capability Induce Mandarin corpus from English corpus Train LM statistics for both recognizers from corpora Develop parsing grammar for Mandarin queries and generation rules for Mandarin responses Not yet completed: Develop domain-specific user simulation capability Generate thousands of dialogues in both languages 3. Train recognizers and users from simulated dialogues Activities over the Last Nine Months:  Activities over the Last Nine Months Translation from English to Mandarin Mainly focused on user queries (as contrasted with responses) Integrating generation-based translation with example-based approach Exploring the use of statistical machine translation Use phrase-based statistical translation framework developed by Phillip Koehn Utilized the formal methods to generate domain-specific parallel corpus in weather query domain Implemented a finite-state transducer version of the decoder and integrated with Galaxy Translation from Mandarin to English Use statistical method to obtain Chinese to English translation capability Explore grammar induction techniques to create parsing grammar for Mandarin queries, towards developing formal methods for Mandarin to English translation Activities over the Last Nine Months, Cont’d:  Activities over the Last Nine Months, Cont’d System Development Upgraded weather harvesting process Upgraded database server to support Postgres in addition to Oracle Improved dialogue management Better handling of meta queries Developed a new GUI interface ovecoming firewall limitations Support automatic checking and correction of typed tone errors Better display of tones as diacritcs Developed a new concatenative speech synthesis capability for high quality translation of user queries spoken in English using Envoice Developed a batchmode capability to process synthetic speech through dialogue interaction to aid system development Activities over the Last Nine Months, Cont’d :  Activities over the Last Nine Months, Cont’d Presentations: Three talks at InStill Workshop in Venice Wang and Seneff: Translation Seneff et al. : LL Systems Peabody et al.: Web based interface for tone acquisition ISCSLP: Seneff et al.: Focused on MuXing system overall SigDial Demo Session Wang and Seneff: Presentation and live demonstration One hour seminar at Microsoft China’s Speech Group One hour seminar at Defense Language Institute in Monterey Demonstrated system to Julian Wheatley, head of Chinese department at MIT and to Henry Jenkins, director of MIT Comparative Media Studies Activities over the Last Nine Months, Cont’d :  Activities over the Last Nine Months, Cont’d Data collection initiatives: Eight subjects have completed Web-based exercise at MIT Two visits by Stephanie Seneff to Defense Language Institute in Monterey California One successful class participation exercise Another attempted but aborted due to power outage Installed Web-based exercise system on computers at MIT Language Lab Julian Wheatley has agreed to support data collection initiatives with students in the MIT Chinese classes Bilingual Recognizer Construction:  Bilingual Recognizer Construction English corpus Two languages compete in common search space Automatically translate existing English corpus into Mandarin Use NL grammar to automatically induce language model for both English and Mandarin recognizers Automatic Grammar Induction:  Automatic Grammar Induction English Sentence Corpus Pairs Once translation ability exists from English to target language, can create reverse system almost effortlessly Utilizes English parse tree and Mandarin generation lexicon to induce Mandarin parse tree Multilingual Spoken Translation Framework:  Multilingual Spoken Translation Framework Common meaning representation: semantic frame Semantic Frame Challenges in Cross-language Generation for Translation:  Challenges in Cross-language Generation for Translation Some expressions have very different syntactic structures in different languages What is your name? 你(you) 叫(call) 什么(what) 名字(name)? I like her. Ella me gusta. 附近(vicinity) 哪儿(where) 有(have) 银行(bank)? Where is a bank nearby? that hotel 那(that) 家(<particle>) 旅馆(hotel) I lost my key. 我(I) 丢(lose) 了(<past tense>) 我的(my) 钥匙(key). Particles (Chinese but not English) Gender (extensive in Spanish) Syntactic features are expressed in many different ways Determiners (English but not Chinese) An Example: English/Chinese:  How long does it take to take a taxi there How long take take taxi there An Example: English/Chinese Function words disappear in Chinese How long does it take to take a taxi there ( take taxi go there need how long ) 坐 出租车 去 那里 要 多久 Sentence structure is very different Verb “go” omitted in English Two instances of “take” have different translations How long need take taxi there How long need take taxi go there Semantic Frame for Example:  Semantic Frame for Example Semantic frame is identical for both inputs, except for missing function words in Mandarin Where necessary, constituent movement is invoked to render the same hierarchical structure English generation predicts missing function words Mandarin generation infers “go” from “destination” predicate Strategies for Achieving High Quality and Robustness:  Strategies for Achieving High Quality and Robustness Interlingua-based translation Maintain consistency of semantic frame representation across different languages whenever possible Seed grammar rules for each new language on English grammar rules Target language dependent generation rules specify constituent order Word sense disambiguation achieved through semantic features Restricted conversational domains (lesson plans) Emphasis on mechanisms to enable rapid porting to new domains and languages Use parsability to assess quality of translation outputs Back off to example-based method when parse fails Schematic of Generation into Mandarin :  Schematic of Generation into Mandarin {c verify :aux “will” :subject “it” :pred {p rain :pred {p locative :prep ‘in” :topic {q city :name “boston” } } :pred {p temporal :topic {q weekday :quanitifier “this” :name “weekend” } } } } Generation-based Translation:  Generation-based Translation Semantic frame serves as interlingua Translation achieved by parsing and generation Use Mandarin grammar to detect potential problems Rejected sentences routed to example-based translation for a second chance Example-based Translation:  Example-based Translation Requires translation pairs and a retrieval mechanism Corpus automatically obtained via the generation-based approach Retrieval based on lean semantic information Encoded as key-value pairs Obtained from semantic frame via simple generation rules Generalizes words to classes (e.g., city name, weekday, etc.) to overcome data sparseness Example-based Translation Procedure:  WEATHER: rain CITY: San Francisco Example-based Translation Procedure Is there any chance of rain in San Francisco? { <CITY> : jiu4 jin1 shan1 } jiu4 jin1 shan1 Key-value string serves as interlingua Translation achieved by parsing and table lookup City name masked during retrieval and recovered in final surface string KV-Chinese Table Evaluation: English to Mandarin Weather Domain:  Evaluation: English to Mandarin Weather Domain Evaluation data Drawn from the publicly available Jupiter weather system Telephone recordings; conversational speech Unparsable utterances (English grammar) were excluded Total of 695 utterances, with 6.5 words per utterance on average System configuration Text input or speech input Recognizer achieved 6.9% word error rate, and 19.0% sentence error rate Generation-based method preferred over example-based method NULL output if both failed Evaluation criteria Yield of each translation method Human judgment of translation quality Spoken Language Translation: Evaluation Results:  Spoken Language Translation: Evaluation Results Recognizer WER was 6.9% Bilingual judge rated translations Example-based translation increased yield by 6% Incorrect translation provided only 2% of the time Often due to recognition errors English paraphrase provides context for errors 13(2%) Multilingual Weather Responses:  clause: weather_event topic: precip_act, name: thunderstorm, num: pl quantifier: some pred: accompanied_by adverb: possibly topic: wind, num: pl, pred: gusty and: precip_act, name: hail English source: Some thunderstorms may be accompanied by gusty winds and hail Frame indexed under wind, rain, storm, and hail Multilingual Weather Responses Stage 1: Drill Exercises:  Stage 1: Drill Exercises Web-based Interface to provide practice in typing queries in the weather domain 10 weather scenarios to be solved using typed pinyin: “Boston, rain, tomorrow” Student given feedback on both query completeness and tone accuracy Separate recording sessions allow user to practice both read and spontaneous spoken queries Recordings will be used to train the system on accented speech Recordings will also be assessed for tone quality The Defense Language Institute in Monterey conducted a successful experiment using this Web-based interface in a class of 30 students We are planning to introduce the exercise in the language laboratory at MIT Lexical Tone Correction:  Lexical Tone Correction Character representation does not explicitly encode tone: 洛杉矶星期一刮风吗? Exploit pinyin to help student acquire tonal knowledge: Diacritic: luò shān jī xīng qī yī guā fēng ma? Numeric: luo4 shan1 ji1 xing1 qi1 yi1 gua1 feng1 ma5? Hypothesis: Errors in typed pinyin reflect inaccurate knowledge of tones luo3 shan1 ji3 xing1 qi2 yi1 gua4 feng2 ma2? Provide explicit feedback about typed tone errors Lexical Tone Correction:  Lexical Tone Correction Exploit some features of Chinese Syllable lexicon is small, approximately 420 unique syllables 5 tones (including neutral tone) Exploit some abilities of TINA NL system Ability to parse weighted word FST using probabilistic models FST normally represents a list of recognizer hypotheses A path through the FST represents the most likely correct parse Given some input Generate FST of single sentence Expand the tones on each syllable Attempt to parse FST Selected path through FST represents corrected tones FST Example: Step 1:  FST Example: Step 1 Step 1: Generate simple FST Given: luo3 shan1 ji3 xing1 qi2 yi1 gua4 feng2 ma2 FST Example: Step 2:  FST Example: Step 2 Step 2: Assign benefit of doubt to items that appear in lexicon Given: luo3 shan1 ji3 xing1 qi2 yi1 gua4 feng2 ma2 Items that do not appear in lexicon are removed. FST Example: Step 3:  FST Example: Step 3 Step 3: Expand each syllable to alternate tones. More compact than specifying each possible sentence variant. Given: luo3 shan1 ji3 xing1 qi2 yi1 gua4 feng2 ma2 FST Example: Step 4:  FST Example: Step 4 Step 4: Remaining probability is uniformly distributed among alternate tones Given: luo3 shan1 ji3 xing1 qi2 yi1 gua4 feng2 ma2 FST Example: Step 5:  FST Example: Step 5 Step 5: Parsing reveals the correct tones Given: luo3 shan1 ji3 xing1 qi2 yi1 gua4 feng2 ma2 Correct: luo4 shan1 ji1 xing1 qi1 yi1 gua1 feng1 ma5 Web interface: Practice Exercise:  Web interface: Practice Exercise Student is prompted for city, time, and event Web interface: Practice Exercise:  Web interface: Practice Exercise Student types in: A question concerning this topic in Mandarin using pinyin OR An English word or phrase for a translation Xing1 qi1 er3 jiu3 jin3 shan1 hui4 bu2 hui4 re1 Web interface: Practice Exercise:  Web interface: Practice Exercise Student is given feedback Web interface:  Web interface Spoken Conversational Interaction:  Weather information domain (rain, snow, wind, temperature, etc.) Initial version configured for American learning Mandarin Recognizer supports both English and Mandarin Seamless language switching English queries are translated into Mandarin Mandarin queries are answered in Mandarin User can ask for a translation into English of the response at any time Uses Mandarin synthesizer provided by DELTA Electronics for responses, Envoice concatenative synthesizer for query translations System can be configured as telephone-only or as telephone augmented with a Web-based gui interface Spoken Conversational Interaction Illustration of Dialogue Interaction:  Illustration of Dialogue Interaction User: Bo1 Shi4 Dun4 ming2 tian1 hui4 xia4 yu3 ma5? (Is it going to rain tomorrow in Boston?) System: Tian1 qi4 yu4 bao4 ming2 tian1 Bo1 shi4 dun4 mei2 you3 yu3. (The forecast calls for no rain tomorrow in Boston) User: (in English) What is the temperature? System: (translates) Qi4 wen1 shi4 duo1 shao3? User: (emulates) Qi4 wen1 shi4 duo1 shao3? System: Bo1 Shi4 Dun4 ming2 tian1 zui4 gao1 qi4 wen1 er4 she4 shi4 du4, ming2 tian1 ye4 jian1, zui4 di4 qi4 wen1 ling2 xia4 wu3 she4 shi4 du4. User: Could you translate that? System: In Boston tomorrow, high 2 degrees Celsius, Tomorrow night, low -5 Celsius. Example Dialogue in Weather Domain:  Example Dialogue in Weather Domain “What is the forecast for San Francisco tomorrow?” System paraphrases request, then answers “Please translate” High quality synthesis for translation using MIT’s Envoice concatenative synthesis framework “Could you repeat that” – system provides translation User emulates in Mandarin and system repeats previous response “Will it rain in London?” “I’m sorry I didn’t understand you.” – response given when it fails to recognize or parse the user query Video Clip:  Video Clip Demo Assessment:  Assessment Phonetic aspects Expand phonological rules to support non-native realizations (e.g., /dh/  /d/ or schwa insertion) Allow realizations of selected phones from native language to compete in recognizer search Tonal aspects (Mandarin) Use tone recognition system (Wang et al., 1998) to score tone productions; highlight worst-scoring words Tabulate frequencies of tone errors in typed inputs (pinyin) Use phase-vocoder techniques (Tang et al., 2001) to repair user’s tone productions by replacing prosodic contour with native speech patterns Fluency measures Word-by-word speaking rate (Chung & Seneff, 1999) Percentage of utterance containing pauses and disfluencies Tone analysis: Native vs Non-Native Mandarin:  Tone analysis: Native vs Non-Native Mandarin Creating pitch contours F0 extracted using algorithm in (Wang and Seneff, 2000) Statistics of each pitch contour over each syllable considered without regard for left or right contexts Normalization Duration normalized by sampling at 10% intervals Pitch normalized according to: Comparisons based on (Wang et al., 2003) Include normalized F0 value, peak, valley, range, peak position, valley position, falling range, and rising range Corpus (from the Defense Language Institute) 2065 utterances from 4 native speakers 4657 utterances from 20 non-native speakers Tonal averages over all syllables: Native Example:  Tonal averages over all syllables: Native Example Slide40:  Tonal averages over all syllables: Non-Native Example Capturing Phonological Errors:  Capturing Phonological Errors Leverage phonological modeling capabilities of SUMMIT Model typical pronunciation errors explicitly Direct and intuitive mapping from linguistic rules Support both within-language and cross-language substitutions Initial experiments completed on Koreans learning English (Kim et al., ICSLP 2004) Phonological rules capture typical problems such as schwa insertion and /dh/ /d/ confusions Best path in alignment used to detect errors Verbal feedback given to student Current research to apply to Americans learning Mandarin Build single recognizer to support both languages Use data-driven approaches to discover most likely cross-language phone substitution errors Explicitly encode such errors in formal phonological rules Side benefit may be improved recognition for English-accented Mandarin Detecting Phonological Errors:  {} dh {} => dh | [dcl] d ; // Becomes an onset stop as in 'they'. No [dh] in Korean phonemes.. {} dd {} => dcl [d [ax]] ; // A vowel may be inserted after a coda consonant (Staccato Rhythm) {CONSONANT} td {CONSONANT} => [tcl] [t] | tcl t [ax]; // No CCC allowed in Korean Detecting Phonological Errors Future Plans:  Future Plans Develop tools to rapidly port to new domains and languages Automatic grammar induction Generic dialogue modeling Simulated dialogue interactions Develop various scoring algorithms for quality assessment of student’s speech Develop high quality synthesis capability for Mandarin translations, for multiple domains of knowledge Collect and transcribe data from language learners and evaluate both system and students Begin with weather domain, our most mature system Extend to other domains once they are better developed Refine all aspects of systems based on collected data

Related presentations


Other presentations created by Lucianna

Nutritional Care of Burns
04. 01. 2008
0 views

Nutritional Care of Burns

spine2 no background
08. 05. 2008
0 views

spine2 no background

banking
14. 04. 2008
0 views

banking

emerging security threats
29. 09. 2007
0 views

emerging security threats

Thunderstorms
03. 10. 2007
0 views

Thunderstorms

i2 traffic shaping
03. 10. 2007
0 views

i2 traffic shaping

bind
07. 10. 2007
0 views

bind

prefix delegation requirement1
09. 10. 2007
0 views

prefix delegation requirement1

dipo
12. 10. 2007
0 views

dipo

Living Things
12. 10. 2007
0 views

Living Things

wnv062904
21. 10. 2007
0 views

wnv062904

latinoamerica
22. 10. 2007
0 views

latinoamerica

Rachinsky
11. 10. 2007
0 views

Rachinsky

Slide presentazione
24. 10. 2007
0 views

Slide presentazione

feynman
16. 10. 2007
0 views

feynman

gt bot
13. 10. 2007
0 views

gt bot

fr summit marginson 230306
30. 10. 2007
0 views

fr summit marginson 230306

Accelerators CZ
15. 11. 2007
0 views

Accelerators CZ

Les Animaux du Zoo
11. 10. 2007
0 views

Les Animaux du Zoo

Rapport Nationale MAROC
23. 10. 2007
0 views

Rapport Nationale MAROC

Grammar essentials
16. 11. 2007
0 views

Grammar essentials

sponge
20. 11. 2007
0 views

sponge

Crans Montana 03 nieuw
15. 10. 2007
0 views

Crans Montana 03 nieuw

Workshop
02. 11. 2007
0 views

Workshop

NSF 12 6 2001
31. 12. 2007
0 views

NSF 12 6 2001

Class8
07. 01. 2008
0 views

Class8

toc wkshp nov03
18. 10. 2007
0 views

toc wkshp nov03

VCT Morocco
24. 10. 2007
0 views

VCT Morocco

NACADA Combined Workshop 11 04
29. 09. 2007
0 views

NACADA Combined Workshop 11 04

sky
13. 11. 2007
0 views

sky

file Kigali Strengthening Local
07. 01. 2008
0 views

file Kigali Strengthening Local

10638221831Maroc MinInt French
23. 10. 2007
0 views

10638221831Maroc MinInt French

ub geographicimagery051 001
27. 09. 2007
0 views

ub geographicimagery051 001

Presentación RR EXPORTA def
23. 10. 2007
0 views

Presentación RR EXPORTA def

prosper
28. 12. 2007
0 views

prosper

HPCN summary 7 5 2007
17. 10. 2007
0 views

HPCN summary 7 5 2007

ammosov
12. 10. 2007
0 views

ammosov

A NEW ENGLISH COURSE Book 3
20. 02. 2008
0 views

A NEW ENGLISH COURSE Book 3

Food Bank of New Jersey
29. 02. 2008
0 views

Food Bank of New Jersey

lewis
19. 10. 2007
0 views

lewis

XC Safety and mentor
03. 04. 2008
0 views

XC Safety and mentor

NA3
07. 04. 2008
0 views

NA3

Civitas Plus2006
18. 03. 2008
0 views

Civitas Plus2006

Ch14 7e
10. 04. 2008
0 views

Ch14 7e

Team2
11. 04. 2008
0 views

Team2

fmla
04. 10. 2007
0 views

fmla

retailcompetition
17. 04. 2008
0 views

retailcompetition

Using ILS
22. 04. 2008
0 views

Using ILS

shaw
16. 03. 2008
0 views

shaw

CSI Presentation 2007
19. 02. 2008
0 views

CSI Presentation 2007

NIST TDT2004
07. 05. 2008
0 views

NIST TDT2004

chapter3v2
15. 10. 2007
0 views

chapter3v2

MEDOPSBOOKFEB01
02. 05. 2008
0 views

MEDOPSBOOKFEB01

BostwPres
02. 05. 2008
0 views

BostwPres

555 Spanish
02. 05. 2008
0 views

555 Spanish

hexapod Shirke
02. 05. 2008
0 views

hexapod Shirke

Lung Expansion 1
02. 05. 2008
0 views

Lung Expansion 1

Aaron
02. 05. 2008
0 views

Aaron

SAP1012
10. 03. 2008
0 views

SAP1012

lesson 4
15. 10. 2007
0 views

lesson 4

2006 APHA
05. 10. 2007
0 views

2006 APHA

probir
30. 03. 2008
0 views

probir

Rauf Presentation NEW
18. 10. 2007
0 views

Rauf Presentation NEW

IAJAPAN
09. 10. 2007
0 views

IAJAPAN

Mr Daisuke Matsunaga
09. 10. 2007
0 views

Mr Daisuke Matsunaga

3 KukaGLBThealthissues
29. 10. 2007
0 views

3 KukaGLBThealthissues

Bernard ANSELMETTI
24. 10. 2007
0 views

Bernard ANSELMETTI

NBII Newark 10 02
21. 10. 2007
0 views

NBII Newark 10 02

MarketingWorkshop 4 22 05rev1
24. 10. 2007
0 views

MarketingWorkshop 4 22 05rev1

FEESTDAGEN
06. 11. 2007
0 views

FEESTDAGEN

trainplanesandautomo biles
13. 03. 2008
0 views

trainplanesandautomo biles

NWA June00
05. 10. 2007
0 views

NWA June00

Panama 2004 Reporte
25. 10. 2007
0 views

Panama 2004 Reporte

SAKURA Yamamoto
25. 03. 2008
0 views

SAKURA Yamamoto

tiner presentation
04. 01. 2008
0 views

tiner presentation

aseancjp
09. 10. 2007
0 views

aseancjp

schools talk
29. 10. 2007
0 views

schools talk

BethkeA
02. 10. 2007
0 views

BethkeA

DeVidtsPresentation
11. 10. 2007
0 views

DeVidtsPresentation

Zhu Zhiyong
16. 10. 2007
0 views

Zhu Zhiyong

pres1 1
22. 10. 2007
0 views

pres1 1

AESC 2005 VERMONT Result
02. 11. 2007
0 views

AESC 2005 VERMONT Result