qa overview cmu

Information about qa overview cmu

Published on October 16, 2007

Author: Abbott

Source: authorstream.com

Content

Overview of Question Answering :  Overview of Question Answering Svetlana Stenchikova “QA with Semantic Roles” - Internship project at AT&T with the mentors: Dilek Hakkani-Tür and Gokhan Tur Presentation Plan:  Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition QA is different from Search:  QA is different from Search Example: Where did Maya people live? :  Example: Where did Maya people live? the Maya people lived in: a large cultural area that archaeologists call Mesoamerica south-central Mexico the region that is now Eastern and Southern Mexico, Guatemala, Belize, El Salvador and Western Honduras small scattered villages of pole and thatch houses QA System: QA Scenarios:  QA Scenarios Limited Domain – Fixed set of documents Information is more reliable but sparse Data preprocessing is feasible Unlimited domain – Web Greater redundancy, but lower reliability On-demand data processing TREC competition Slide6:  QA System Architecture Related Work:  Related Work Insight(Russia): extensive use of patterns START (MIT): first QA system on the web. Uses Knowledge Annotation Technique: annotating resources with metadata Use specific data source for different queries Slide8:  Related Work AskMSR: count 1-, 2-, 3-grams on the snippet outputs, finds the most frequent item. Use answer tiling to generate longer answers Insight(Russia) START (MIT) Slide9:  Related Work AskMSR Insight(Russia) LCC: Extract Logical Form from questions and answers use ontology to “paraphrase” use logic provers for answer verification Search for answer in a paragraph instead of a sentence START (MIT). Presentation Plan:  Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition Slide11:  QA System Architecture Who won the Nobel prize in literature in 1988? “won the Nobel prize in literature in 1988” http://www.nytimes.com/books/99/04/18/specials/rushdie-rally.html http://www.newyorker.com/printables/fact/040712fa_fact1 … Naguib Mahfouz won the nobel prize in literature in 1988 Egyptian writer won the nobel prize in literature in 1988 A distinguished Egyptian writer Naguib Mahfouz won the nobel prize in literature in 1988 Naguib Mahfouz Egyptian writer 2/3 1/3 Candidate Sentences – Actual Scenario:  Candidate Sentences – Actual Scenario The veritable terrorism of which he is a target is unjustifiable , indefensible , " wrote Naguib Mahfouz , the Egyptian who won the Nobel Prize in Literature in 1988 Mahfouz , who won the Nobel Prize in Literature in 1988 , is known for his intimate stories of Egyptian life … Slide13:  Phrase Extraction and Search Methods Search: find candidate documents Exact Phrase Conjunction of sub-phrases (inexact phrase) Slide14:  Phrase Extraction and Search: Exact Phrase If a question contains an auxiliary verb (is/was/were/are/did/do/does): drop all words between the 'wh' word and the auxiliary match the tense of the predicate with the auxiliary verb. When did Vesuvius last erupt? ~ “Vesuvius last erupted” What state does Martha Stewart live in ~ “Martha Stewart lives in” Slide15:  Phrase Extraction and Search : Inexact Phrase SRL-parse the question, identifying the argument phrases: Who invented the electric guitar? [ARG0 Who] [TARGET invented] [ARG1 the electric guitar] Search for “invented” AND “the electric guitar” Searching: Use Google search API Slide16:  Sentence Extraction Methods Sentence Extraction: finding candidate sentences Exact phrase Conjunction of phrases Predicate Evaluation of Search and Sentence Extraction:  Evaluation of Search and Sentence Extraction Accuracy: % of candidate sentences containing a correct answer Coverage: % of the queries containing at least one correct answer in the set of candidate sentences Slide18:  SRL can improve the result ONLY when a correct answer is present in candidate sentences. Future work: Improve Search/extraction Using synonyms for search and extractions Back-off on prepositional clauses Future Work on Sentence Extraction Baseline Answer Extraction:  Baseline Answer Extraction Single template: Answer appears on the left of the right side of the search phrase Question: Who invented electric guitar? Phrase: “invented the electric guitar” Pattern: […] invented the electric guitar Question: When was Nostradamus born? Phrase: “Nostradamus was born” Pattern : Nostradamus was born in […] SRL Architecture:  SRL Architecture Q: Who created the comic strip Garfield? Predicate: created Argument type: ARG0 SRL Parse: [ARG0 Who] [TARGET created ] [ARG1 the comic strip Garfield] Task: Find ARG0 of the TARGET created Semantic Role module Semantic Role module Slide21:  SRL Architecture Q: Who created the comic strip Garfield? Predicate: created Argument type: ARG0 SRL Candidate Sentence: Garfield is [ARG1 a popular comic strip] [TARGET created ] [ARG0 by Jim Davis] featuring the cat Garfield ... Extracted Answers: by Jim Davis Semantic Role module Semantic Role module Automatic Evaluation: Mean Reciprocal Ranking (MRR) and precision of correct answer:  Automatic Evaluation: Mean Reciprocal Ranking (MRR) and precision of correct answer 0.24 => 0.35 is a statistically significant improvement MRR: question gets score = 1/N where N is the index of the first correct answer Use 5 top answers Precision(% questions with first answer being correct) Measure of Conciseness and Irrelevant Information in Correct Answers:  Measure of Conciseness and Irrelevant Information in Correct Answers * This experiment was done manually Presentation Plan:  Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition TREC competition:  TREC competition TREC competitions QA track ~ Since 1999 Set of factoid/definition/context questions. Data Set 3 GB Slide26:  TREC history TREC 2004 - 2006 main task:  TREC 2004 - 2006 main task Define a topic target for a set of questions: Person George Bush Thing Shiite Organization Amway Event return of Hong Kong to Chinese sovereignty (Event was introduced in 2005) Question Types: List Factoid Other The answer to the "Other" question is to be interesting information about the target that is not covered by the preceding questions in the series. TREC Main Task Question Example:  TREC Main Task Question Example target text="Shiite“ FACTOID    Who was the first Imam of the Shiite sect of Islam?  FACTOID  Where is his tomb? FACTOID    What was this person's relationship to the Prophet Mohammad? FACTOID    Who was the third Imam of Shiite Muslims? FACTOID  When did he die? FACTOID  What portion of Muslims are Shiite? LIST What Shiite leaders were killed in Pakistan? OTHER    Other Scoring :  Scoring 1/3*factoid-score + 1/3*list-score + 1/3*other-score. Factoid answers are judged as: "incorrect", "unsupported", "non-exact", "locally correct", or "globally correct" factoid-score= % "globally correct" Factoid and List Answer Scoring:  Factoid and List Answer Scoring Factoid: count an answer as correct if the answer string consists of exactly the right answer (and supported by the document). Report accuracy - % of correct answers List: An individual instance is interpreted as for factoid questions IR = # instances judged correct & distinct/|final answer set| IP = # instances judged correct & distinct/# instances returned F = (2*IP*IR)/(IP+IR) Other Question Scoring:  Other Question Scoring Inspired by: Jimmy Lin and Dina Demner-Fushman. Will Pyramids Built of Nuggets Topple Over? the assessor creates a list of acceptable information nuggets from the union of the returned responses and the information discovered during question development Assessor marks “vital” and “Okay” nuggets present in the system’s answer Precision and recall are computed as a function of # VITAL and # OKAY nuggets in response TREC 2005 “Factoid” Questions Evaluation:  TREC 2005 “Factoid” Questions Evaluation TREC 2005 “List” Questions Results:  TREC 2005 “List” Questions Results TREC 2005 “other” Questions Results:  TREC 2005 “other” Questions Results Complex Interactive QA Track:  Complex Interactive QA Track Questions were created by 4 military analysts Evaluated the same way as “other” in Main QA task In 2005 competition scores range from .061 - .276 Complex Interactive QA Track:  Complex Interactive QA Track What evidence is there for transport of [goods] from [entity] to [entity]? What [relationship] exist between [entity] and [entity]? where [relationship] is a element of {"financial relationships", "organizational ties", "familial ties", "common interests"} What influence/effect do(es) [entity] have on/in [entity]? What is the position of [entity] with respect to [issue]? Is there evidence to support the involvement of [entity] in [event/entity]? Question Example:  Question Example   <topic num="1"> <template id="1"> What evidence is there for transport of [drugs] from [Bonaire] to [the United States]? </template> <narrative> The analyst would like to know of efforts made to discourage narco traffickers from using Bonaire as a transit point for drugs to the United States. Specifically, the analyst would like to know of any efforts by local authorities as well as the international community. </narrative>   </topic> References:  References http://www.umiacs.umd.edu/~jimmylin/ciqa/guidelines.html Overview of the TREC 2005 Question Answering Track E.M. Voorhees, H.T. Dang, NIST http://trec.nist.gov/act_part/tracks/qa/qa.06.guidelines.html THANK YOU:  THANK YOU

Related presentations


Other presentations created by Abbott

Fantastic Pictures
31. 07. 2007
0 views

Fantastic Pictures

jovanovi
02. 05. 2008
0 views

jovanovi

ours2
30. 04. 2008
0 views

ours2

golding
28. 04. 2008
0 views

golding

SandersD
22. 04. 2008
0 views

SandersD

MasterclassBasicsOfH edging
18. 04. 2008
0 views

MasterclassBasicsOfH edging

Energy Management Basics
17. 04. 2008
0 views

Energy Management Basics

jpn ma
13. 04. 2008
0 views

jpn ma

0802 Roderick Deane
10. 04. 2008
0 views

0802 Roderick Deane

Athirapally Vazhachaal Falls
26. 09. 2007
0 views

Athirapally Vazhachaal Falls

Lecture7
10. 10. 2007
0 views

Lecture7

11 GUIDE presentation Porvoo 10
19. 10. 2007
0 views

11 GUIDE presentation Porvoo 10

Egyptian Quiz
21. 10. 2007
0 views

Egyptian Quiz

AGE200524122005
24. 10. 2007
0 views

AGE200524122005

Henry VIII wwt bam
21. 08. 2007
0 views

Henry VIII wwt bam

DOE LNG FORUM 14
07. 11. 2007
0 views

DOE LNG FORUM 14

matzgdh6
22. 10. 2007
0 views

matzgdh6

BdReviewFINAL2006 lq
16. 11. 2007
0 views

BdReviewFINAL2006 lq

session2
20. 11. 2007
0 views

session2

lecture26 fall 2003
19. 11. 2007
0 views

lecture26 fall 2003

WWF
26. 11. 2007
0 views

WWF

Chocolate 01
03. 10. 2007
0 views

Chocolate 01

MARE 494 Lecture 12
12. 10. 2007
0 views

MARE 494 Lecture 12

ErnestoRivera
22. 10. 2007
0 views

ErnestoRivera

InherentSafety Westray
10. 12. 2007
0 views

InherentSafety Westray

tema1
24. 10. 2007
0 views

tema1

Libecap powerpoint
04. 01. 2008
0 views

Libecap powerpoint

Assam
05. 01. 2008
0 views

Assam

Samuels Mol Pharm 11 03
24. 02. 2008
0 views

Samuels Mol Pharm 11 03

Stock Market Game
24. 02. 2008
0 views

Stock Market Game

nv specialitypp
29. 10. 2007
0 views

nv specialitypp

104 04MagnConst
13. 11. 2007
0 views

104 04MagnConst

clingan
04. 03. 2008
0 views

clingan

EARNIN x007E 1 2
23. 10. 2007
0 views

EARNIN x007E 1 2

aes
13. 03. 2008
0 views

aes

Semprini
16. 03. 2008
0 views

Semprini

KPMG PresentationSMEfinal
26. 03. 2008
0 views

KPMG PresentationSMEfinal

temple of Zeus
21. 08. 2007
0 views

temple of Zeus

Australia 2 March 20051
28. 09. 2007
0 views

Australia 2 March 20051

maillistnews
05. 10. 2007
0 views

maillistnews

feb 5
27. 09. 2007
0 views

feb 5

ykis06 K F Liu
15. 11. 2007
0 views

ykis06 K F Liu

schreiber
12. 10. 2007
0 views

schreiber

Pisani
30. 10. 2007
0 views

Pisani

200752414331362
11. 10. 2007
0 views

200752414331362

decisiontheoryreview
04. 01. 2008
0 views

decisiontheoryreview

a Macroecologia Alimentar 2
22. 10. 2007
0 views

a Macroecologia Alimentar 2

praesentationschule
02. 11. 2007
0 views

praesentationschule

Wireless Data Business
18. 06. 2007
0 views

Wireless Data Business

WHA userguide Customer2006
18. 06. 2007
0 views

WHA userguide Customer2006

wales
18. 06. 2007
0 views

wales

VON04s I2
18. 06. 2007
0 views

VON04s I2

UTeV Rick Field 12 14 06
18. 06. 2007
0 views

UTeV Rick Field 12 14 06

UDDI Overview Presentation
18. 06. 2007
0 views

UDDI Overview Presentation

2006 02 28T124652
25. 03. 2008
0 views

2006 02 28T124652

SoprolecheAgo 2005
23. 10. 2007
0 views

SoprolecheAgo 2005

ZadehTalk
18. 10. 2007
0 views

ZadehTalk

ihepccc hepix
24. 10. 2007
0 views

ihepccc hepix

jedwards
02. 10. 2007
0 views

jedwards

Wireless Mobile Platform OAEP
15. 06. 2007
0 views

Wireless Mobile Platform OAEP

Taverna Workbench
15. 06. 2007
0 views

Taverna Workbench

Job Services with Genius Portal
15. 06. 2007
0 views

Job Services with Genius Portal

MSN screenshots
15. 06. 2007
0 views

MSN screenshots

Caltech, Peck Fest Jan 05
15. 06. 2007
0 views

Caltech, Peck Fest Jan 05

rslt3218
17. 10. 2007
0 views

rslt3218

6 26
17. 10. 2007
0 views

6 26

vp sigmetrics 04 adversaries
18. 06. 2007
0 views

vp sigmetrics 04 adversaries

calcagno
22. 10. 2007
0 views

calcagno

LegoDesign
31. 12. 2007
0 views

LegoDesign

Ted Hanss Internet2
15. 06. 2007
0 views

Ted Hanss Internet2

Mary and Darnley
21. 08. 2007
0 views

Mary and Darnley

1 partie
17. 10. 2007
0 views

1 partie

Techmission Safe Families
15. 06. 2007
0 views

Techmission Safe Families

Camille Page CrocodileGuy MK8
11. 10. 2007
0 views

Camille Page CrocodileGuy MK8

Salandayia
04. 10. 2007
0 views

Salandayia

B4 Hamilton
03. 01. 2008
0 views

B4 Hamilton

SEC062804
25. 10. 2007
0 views

SEC062804