Lec1 intro

Information about Lec1 intro

Published on November 21, 2007

Author: The_Rock

Source: authorstream.com

Content

CS598 DNR FALL 2005 Machine Learning in Natural Language:  CS598 DNR FALL 2005 Machine Learning in Natural Language Dan Roth University of Illinois, Urbana-Champaign [email protected] http://L2R.cs.uiuc.edu/~danr Comprehension:  Comprehension (ENGLAND, June, 1989) - Christopher Robin is alive and well. He lives in England. He is the same person that you read about in the book, Winnie the Pooh. As a boy, Chris lived in a pretty home called Cotchfield Farm. When Chris was three years old, his father wrote a poem about him. The poem was printed in a magazine for others to read. Mr. Robin then wrote a book. He made up a fairy tale land where Chris lived. His friends were animals. There was a bear called Winnie the Pooh. There was also an owl and a young pig, called a piglet. All the animals were stuffed toys that Chris owned. Mr. Robin made them come to life with his words. The places in the story were all near Cotchfield Farm. Winnie the Pooh was written in 1925. Children still love to read about Christopher Robin and his animal friends. Most people don't know he is a real person who is grown now. He has written two books of his own. They tell what it is like to be famous. 1. Who is Christopher Robin? 2. When was Winnie the Pooh written? 3. What did Mr. Robin do when Chris was three years old? 4. Where did young Chris live? 5. Why did Chris write two books of his own? What we Know: Ambiguity Resolution:  Illinois’ bored of education board ...Nissan Car and truck plant is … …divide life into plant and animal kingdom (This Art) (can N) (will MD) (rust V) V,N,N The dog bit the kid. He was taken to a veterinarian a hospital Tiger was in Washington for the PGA Tour What we Know: Ambiguity Resolution Comprehension:  Comprehension (ENGLAND, June, 1989) - Christopher Robin is alive and well. He lives in England. He is the same person that you read about in the book, Winnie the Pooh. As a boy, Chris lived in a pretty home called Cotchfield Farm. When Chris was three years old, his father wrote a poem about him. The poem was printed in a magazine for others to read. Mr. Robin then wrote a book. He made up a fairy tale land where Chris lived. His friends were animals. There was a bear called Winnie the Pooh. There was also an owl and a young pig, called a piglet. All the animals were stuffed toys that Chris owned. Mr. Robin made them come to life with his words. The places in the story were all near Cotchfield Farm. Winnie the Pooh was written in 1925. Children still love to read about Christopher Robin and his animal friends. Most people don't know he is a real person who is grown now. He has written two books of his own. They tell what it is like to be famous. 1. Who is Christopher Robin? 2. When was Winnie the Pooh written? 3. What did Mr. Robin do when Chris was three years old? 4. Where did young Chris live? 5. Why did Chris write two books of his own? An Owed to the Spelling Checker:  I have a spelling checker, it came with my PC It plane lee marks four my revue Miss steaks aye can knot sea. Eye ran this poem threw it, your sure reel glad two no. Its vary polished in it's weigh My checker tolled me sew. A checker is a bless sing, it freeze yew lodes of thyme. It helps me right awl stiles two reed And aides me when aye rime. Each frays come posed up on my screen Eye trussed to bee a joule... introduction An Owed to the Spelling Checker Intelligent Access to Information:  Intelligent Access to Information Access free form text News articles; reports: maintenance, projects,…. E-mail ; web data Mixed form information: layout intensive (lists, tables,databases…) As if it was a data base Ability to identify the semantics of the text Specific Tasks: Basic recognition, categorization & tagging tasks; semantic analysis; semantic integration; textual entailment,…. Done within and across documents Techniques: Machine Learning and Inference Intelligent Access to Information:Tasks:  Intelligent Access to Information:Tasks Named Entity (Semantic Categorization): Identifying names of entities and concepts in the text Semantic Relations: Identifying relations between entities in documents Cross-Document Entities Identification and Tracing Robust Reading of Text; overcome variability in writing Temporal integration of Information Tracking of entities along time; information integration; change. Question Answering [JFK was busy; the parking lots were full] LOC [Dr. ABC joined Microsoft, Redmond and will lead the SearchIt project.] [The JFK problem; The Michael Jordan Problem [Dr. ABC joined Google to save the AnswerIt project.] Demo:  Demo Screen shot from a CCG demo http://L2R.cs.uiuc.edu/~cogcomp More work on this problem: Scaling up Integration with DBs Temporal Integration/Inference …… Understanding Questions:  Understanding Questions What is the question asking? (different from Googling) Beyond finding candidate passages; choose the right one. Q: What is the fastest automobile in the world? A1: …will stretch Volkswagen’s lead in the world’s fastest growing vehicle market. Demand for cars is expected to soar A2: …the Jaguar XJ220 is the dearest (415,000 pounds), fastest (217mph) and most sought after car in the world. Context: News articles (SJM, LAT,…) And, what if the answers require aggregation,… Not So Easy:  Not So Easy Question Processing:  Question Processing Requires several levels of analysis Syntactic/Functional Analysis (+lexical information) Semantic Annotation Global Analysis: Determining the type of the question Determining the type of the expected answer Determine properties (constraints) on the answer An abstract representation of the question All these are viewed as learning* problems Tools :  Tools A collection of tools that are essential for any intelligent use of text… Robust text analysis tools Tokenization; POS tagging; Shallow parsing Name Entity Classifiers people; locations; organizations; transpotration; materials… Information Extraction functional phrases (e.g., job descriptions; acquisitions) Relations/Event recognizers born_in(A,B); capital_of(C,D); killed(A,B) Role of the Tutorial…. Classification: Ambiguity Resolution:  Illinois’ bored of education [board] Nissan Car and truck plant; plant and animal kingdom (This Art) (can N) (will MD) (rust V) V,N,N The dog bit the kid. He was taken to a veterinarian; a hospital Tiger was in Washington for the PGA Tour  Finance; Banking; World News; Sports Important or not important; love or hate Classification: Ambiguity Resolution Classification:  The goal is to learn a function f: X Y that maps observations in a domain to one of several categories. Task: Decide which of {board ,bored } is more likely in the given context: X: some representation of: The Illinois’ _______ of education met yesterday… Y: {board ,bored } Typical learning protocol: Observe a collection of labeled examples (x,y) 2 X £ Y Use it to learn a function f:XY that is consistent with the observed examples, and (hopefully) performs well on new, previously unobserved examples. Classification Classification is Well Understood:  Theoretically: generalization bounds How many example does one need to see in order to guarantee good behavior on previously unobserved examples. Algorithmically: good learning algorithms for linear representations. Can deal with very high dimensionality (106 features) Very efficient in terms of computation and # of examples. On-line. Key issues remaining: Learning protocols: how to minimize interaction (supervision); how to map domain/task information to supervision; semi-supervised learning; active learning. ranking. What are the features? No good theoretical understanding here. Classification is Well Understood Output Data:  before after word(an) tag(DT) word(intelligence) tag(NN) word(Iraqi) tag(JJ) before before before ... ... after after after end begin Output Data Learn this Structure (Many dependent Classifiers; Finding best coherent structure  INFERENCE) Map Structures (Determine equivalence or entailment between structures  INFERENCE) Extract Features from this structure  INFERENCE Output Data:  before after word(an) tag(DT) word(intelligence) tag(NN) word(Iraqi) tag(JJ) before before before ... ... after after after end begin Output Data Semantic Parse (Semantic Role Labelling):  Screen shot from a CCG demo http://L2R.cs.uiuc.edu/~cogcomp Semantic Parse (Semantic Role Labelling) Textual Entailment:  By “textually entailed” we mean: most people would agree that one sentence implies the other. Textual Entailment WalMart defended itself in court today against claims that its female employees were kept out of jobs in management because they are women WalMart was sued for sexual discrimination Entails Subsumed by  Why Textual Entailment?:  A fundamental task that can be used as a building block in multiple NLP and information extraction applications Has multiple direct applications Why Textual Entailment? Examples:  A key problem in natural language understanding is to abstract over the inherent syntactic and semantic variability in natural language. Multiple tasks attempt to do just that. Relation Extraction: Dole’s wife, Elizabeth, is a native of Salisbury, N.C.  Elizabeth Dole was born in Salisbury, N.C Information Integration (Data Bases) Different database schemas represent the same information under different titles. Information retrieval: Multiple issues, from variability in the query and target text, to relations Summarization; Paraphrasing Multiple techniques can be applied; all are entailment problems. Examples You may disagree with the truth of this statement; and you may infer also that: the presidential candidate’s wife was born in N.C. Question Answering:  Given: Q: Who acquired Overture? Determine: A: Eyeing the huge market potential, currently led by Google, Yahoo took over search company Overture Services Inc last year. Question Answering Eyeing the huge market potential, currently led by Google, Yahoo took over search company Overture Services Inc last year Yahoo acquired Overture Entails Subsumed by  (and distinguish from other candidates) Direct Application: Semantic Verification:  Direct Application: Semantic Verification (and distinguish from other candidates) Given: A long contract that you need to ACCEPT Determine: Does it satisfy the 3 conditions that you really care about? ACCEPT? Role of Learning:  Role of Learning “Solving” a natural language problem requires addressing a wide variety of questions. Learning is at the core of any attempt to make progress on these questions. Learning has multiple purposes: Knowledge Acquisition Classification/Context sensitive disambiguation integration of various knowledge sources to ensure robust behavior Why Natural Language?:  Main channel of Communication Knowledge Acquisition Important for Cognition and Engineering perspectives A grand application: Human computer interaction. Language understanding and generation Knowledge acquisition NL interface to complex systems Querying NL databases Reducing information overload Why Natural Language? introduction Why Learning in Natural Language?:  Challenging from a Machine Learning perspective There is no significant aspect of natural language that can be studied without giving learning a principle role.. Language Comprehension: a large scale phenomenon in terms of both knowledge and computation Requires an integrated approach: need to solve problems in learning, inference, knowledge representation… There is not “cheating”: no toy problems. Why Learning in Natural Language? This Course:  This Course There are many topics to cover. A very active field of research, many ideas are floating around, most of which will not stay. Rather than covering problems - we will cover some of the main ideas and techniques. Attempt to abstract away from specific works and understand the main paradigms. Move towards: Beyond Classification Knowledge Representation and Inference This Course:  This Course Representation-less Approaches Statistics Paradigms Generative and Discriminative Understanding why things work Classification: Learning Algorithms Generative and Discriminative algorithms The ubiquitousness of Linear Representations Features and Kernels Inference Generative models; Conditional Models Inference with Classifiers Constraint satisfaction Structural Mappings (translation) Problems Knowledge Acquisition Multi words? Classification Classification of Verbs? Co-reference? Inferring Sequential Structure (Semantic) Parsing Story Comprehension Answer Selection The Course Plan:  The Course Plan Introduction Why Learning; Learning vs. Statistic Learning Paradigms Theory of Generalization Generative Models LSQ ( Probabilistic Approaches Work) Power of Generative Models Modeling HMM, K-Means, EM; Semi-Sup;ME Discriminatory Algorithm Classification & Inference Linear Learning Algorithms Learning Structured Representations Inference: putting things together Representation & Inference Sequential and General structures Structure Mapping Features Feature extraction languages Kernels (over structures) Sequential Structures Verb Classifications ? Parsing; NE Story Comprehension Representation-Less Approaches Statistics & Information Theory MultiWords ? More Detailed Plan (I):  Introduction to Natural Language Learning Why is it difficult ? Statistics vs. Learning When do we need learning? Examples of problems Statistics and Information Theory Corpus based work: data and tasks. Learning Paradigms PAC Learning Bayesian Learning Examples More Detailed Plan (I) More Detailed Plan (II):  Learning Algorithms Examples General Paradigm: feature based representation Linear functions On line algorithms: additive/multiplicative update Decision Lists; TBL Memory Based Bayesian Methods Naïve Bayes HMMs (Predictions and model learning) Max Entropy LSQ: why do probabilistic algorithms work? More Detailed Plan (II) More Detailed Plan (III):  Relaxing Supervision EM Semi supervised learning: co-learning vs. selection Inference: Sequential Models HMMs (Predictions and model learning) HMMs (with Classifiers), PMMs Constraint Satisfaction/ ad hoc methods Inference: Complex Models Inference as constrained optimization Parsing Structural Mapping Generative vs Discriminative Projects More Detailed Plan (III) Who Are You?:  Who Are You? Undergrads? Ph.D students? Post Ph.D? Background: Natural Language Learning Algorithms/Theory of Computation Survey Expectations:  Expectations Interaction is important! Please, ask questions and make comments Read; Present; Work on projects. Independence: (do, look for, read) more than surface level Rigor: advanced papers will require more Math than you know… Critical Thinking: don’t simply believe what’s written; criticize and offer better alternatives Next Time:  Next Time Examine some of the philosophical themes and leading ideas that motivate statistical approaches to linguistics and natural language and to Begin exploring what can be learned by looking at statistics of texts.

Related presentations


Other presentations created by The_Rock

airlines
30. 03. 2008
0 views

airlines

tradepolicy
22. 04. 2008
0 views

tradepolicy

AVCA CHRIS MWEBESA
13. 04. 2008
0 views

AVCA CHRIS MWEBESA

wolf032806
10. 04. 2008
0 views

wolf032806

UEI SI2006 CitiesIntro
07. 04. 2008
0 views

UEI SI2006 CitiesIntro

Wisenews
28. 03. 2008
0 views

Wisenews

Mogi Tsinghua
27. 03. 2008
0 views

Mogi Tsinghua

2007070302122666487
26. 03. 2008
0 views

2007070302122666487

networking 101
19. 06. 2007
0 views

networking 101

Bowen Family Systems Theory
19. 02. 2008
0 views

Bowen Family Systems Theory

cipher machines
01. 01. 2008
0 views

cipher machines

huwang
29. 09. 2007
0 views

huwang

amart3
02. 10. 2007
0 views

amart3

Finland Nokia
07. 09. 2007
0 views

Finland Nokia

SINUS Milieu stud
12. 10. 2007
0 views

SINUS Milieu stud

AAS SD05 present
29. 11. 2007
0 views

AAS SD05 present

Germany Voting
27. 08. 2007
0 views

Germany Voting

disaster 2001
27. 08. 2007
0 views

disaster 2001

Bergstrom
27. 08. 2007
0 views

Bergstrom

workshop24 2
27. 08. 2007
0 views

workshop24 2

History of electric charge
20. 11. 2007
0 views

History of electric charge

Debra Lieberman
22. 11. 2007
0 views

Debra Lieberman

Reading Genres
27. 08. 2007
0 views

Reading Genres

Plant Establishment
02. 01. 2008
0 views

Plant Establishment

SPACE
03. 01. 2008
0 views

SPACE

IntroductionTo OWL
04. 01. 2008
0 views

IntroductionTo OWL

HBCU Deans Seminar 2005
01. 10. 2007
0 views

HBCU Deans Seminar 2005

genX
27. 09. 2007
0 views

genX

499HodoCuba04
11. 12. 2007
0 views

499HodoCuba04

ext ref uncertainty
27. 08. 2007
0 views

ext ref uncertainty

Intro HTML
09. 10. 2007
0 views

Intro HTML

Ransom Pollen Drift in Wheat
04. 10. 2007
0 views

Ransom Pollen Drift in Wheat

overview norway with notes final
07. 09. 2007
0 views

overview norway with notes final

2004 CFS Power Point for LEPC
26. 02. 2008
0 views

2004 CFS Power Point for LEPC

fps ml talk
28. 02. 2008
0 views

fps ml talk

11979
29. 02. 2008
0 views

11979

presentation cons 2007 e
04. 03. 2008
0 views

presentation cons 2007 e

knox trail proposal
10. 03. 2008
0 views

knox trail proposal

wwired scare
03. 01. 2008
0 views

wwired scare

Banff
13. 03. 2008
0 views

Banff

Technological Prerequisites
18. 03. 2008
0 views

Technological Prerequisites

SKH TOURCAN POWERPOINT
21. 03. 2008
0 views

SKH TOURCAN POWERPOINT

Un Arco Iris para ti 2066
21. 06. 2007
0 views

Un Arco Iris para ti 2066

Diarréia Crônica em AIDS
28. 12. 2007
0 views

Diarréia Crônica em AIDS

liebeversicherung
19. 06. 2007
0 views

liebeversicherung

Lab49 Algo v6
19. 06. 2007
0 views

Lab49 Algo v6

SWR0311
19. 06. 2007
0 views

SWR0311

Tibetan Dali Lama questions
19. 06. 2007
0 views

Tibetan Dali Lama questions

15828233
03. 10. 2007
0 views

15828233

strand
19. 06. 2007
0 views

strand

Yo quisiera ser 1953
21. 06. 2007
0 views

Yo quisiera ser 1953

Sin vicios para matrimonio 2008
21. 06. 2007
0 views

Sin vicios para matrimonio 2008

Lesson7 OverheadT ransparencies
19. 06. 2007
0 views

Lesson7 OverheadT ransparencies

Profesiones 1950
21. 06. 2007
0 views

Profesiones 1950

slides oct04
19. 06. 2007
0 views

slides oct04

Suarez pres
28. 09. 2007
0 views

Suarez pres

140103
10. 10. 2007
0 views

140103

FIcase Oslo workshop
07. 09. 2007
0 views

FIcase Oslo workshop

two years
19. 06. 2007
0 views

two years

MEDITARE
19. 06. 2007
0 views

MEDITARE

Se necesita amor 1955
21. 06. 2007
0 views

Se necesita amor 1955

OAG Sinha
19. 06. 2007
0 views

OAG Sinha

spit
19. 06. 2007
0 views

spit