TextMining 06

Information about TextMining 06

Published on October 3, 2007

Author: Barbara

Source: authorstream.com

Content

Text Mining: Finding Nuggets in Mountains of Textual Data:  Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Dijrre, Peter Gerstl, Roland Seiffert Presented by Drew DeHaas Outline:  Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions Motivation:  Motivation A large portion of a company’s data is unstructured or semi-structured Letters Emails Phone recordings Contracts Technical documents Patents Web pages Articles Motivation:  Motivation Rapid processing of large document collections Speed! Automation of tasks Objective analysis Typical Applications:  Typical Applications Summarizing documents Discovering/monitoring relations among people, places, organizations, etc Organizing documents by content Indexing for search and retrieval Retrieving documents by content Outline:  Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions Methodology: Challenges:  Methodology: Challenges Information is in unstructured textual form Natural language interpretation is difficult & complex task! (not fully possible) Text mining deals with huge collections of documents Methodology: Two Aspects:  Methodology: Two Aspects Knowledge Discovery Extraction of codified information Mining proper; determining some structure Information Distillation Analysis of feature distribution Two Text Mining Approaches:  Two Text Mining Approaches Extraction Extraction of codified information from single document Analysis Analysis of the features to detect patterns, trends, etc, over whole collections of documents Comparison with Data Mining:  Comparison with Data Mining Data mining Identify data set(s) Select features manually Prepare data Analyze distribution Text mining Identify documents Extract features Select features (automatically) Prepare data Analyze distribution IBM Intelligent Miner for Text:  IBM Intelligent Miner for Text IBM introduced product in 1998 SDK with: Feature extraction, clustering, categorization, and more Traditional components (search engine, etc) No longer available? The rest of the paper describes text mining methodology of Intelligent Miner. Outline:  Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions Feature Extraction:  Feature Extraction Recognize and classify “significant” vocabulary items from the text Categories of vocabulary Proper names Multiword terms Abbreviations Relations Other useful things: numerical forms of numbers, percentages, money, etc Canonical Form Examples:  Canonical Form Examples Normalize numbers, money Four = 4, five-hundred dollar = $500 Conversion of date to normal form Morphological variants Drive, drove, driven = drive Proper names and other forms Mr. Johnson, Bob Johnson, The author = Bob Johnson Feature Extraction Approach:  Feature Extraction Approach Linguistically motivated heuristics Pattern matching Limited lexical information (part-of-speech) Avoid analyzing with too much depth Does not use too much lexical information No in-depth syntactic or semantic analysis Feature Extraction Example:  Feature Extraction Example Disambiguating Proper Names (Nominator Program) Apply heuristics to strings, instead of interpreting semantics The unit of context for extraction is a document. The heuristics represent English naming conventions Advantages to IBM’s approach:  Advantages to IBM’s approach Processing is very fast (helps when dealing with huge amounts of data) Heuristics work reasonably well Generally applicable to any domain Outline:  Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions Clustering:  Clustering Fully automatic process Documents are grouped according to similarity of their feature vectors Each cluster is labeled by a listing of the common terms/keywords Good for getting an overview of a document collection Two Clustering Engines:  Two Clustering Engines Hierarchical clustering Orders the clusters into a tree reflecting various levels of similarity Binary relational clustering Flat clustering Relationships of different strengths between clusters, reflecting similarity Clustering Model:  Clustering Model Categorization:  Categorization Assigns documents to preexisting categories Classes of documents are defined by providing a set of sample documents. Training phase produces “categorization schema” Documents can be assigned to more than one category If confidence is low, document is set aside for human intervention Categorization Model:  Categorization Model Outline:  Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions Applications:  Applications Customer Relationship Management application provided by IBM Intelligent Miner for Text called “Customer Relationship Intelligence” “Help companies better understand what their customers want and what they think about the company itself” Customer Intelligence Process:  Customer Intelligence Process Take as input body of communications with customer Cluster the documents to identify issues Characterize the clusters to identify the conditions for problems Assign new messages to appropriate clusters Customer Intelligence Usage:  Customer Intelligence Usage Knowledge Discovery Clustering used to create a structure that can be interpreted Information Distillation Refinement and extension of clustering results Interpreting the results Tuning of the clustering process Selecting meaningful clusters Outline:  Outline Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions Conclusion:  Conclusion This paper introduced text mining and how it differs from data mining proper. Focused on the tasks of feature extraction and clustering/categorization Presented an overview of the tools/methods of IBM’s Intelligent Miner for Text Exam Question #1:  Exam Question #1 Name an example of each of the two main classes of applications of text mining. Knowledge Discovery: Discovering a common customer complaint in a large collection of documents containing customer feedback. Information Distillation: Filtering future comments into pre-defined categories Exam Question #2:  Exam Question #2 How does the procedure for text mining differ from the procedure for data mining? Adds feature extraction phase Infeasible for humans to select features manually The feature vectors are, in general, highly dimensional and sparse Exam Question #3:  Exam Question #3 In the Nominator program of IBM’s Intelligent Miner for Text, an objective of the design is to enable rapid extraction of names from large amounts of text. How does this decision affect the ability of the program to interpret the semantics of text? Does not perform in-depth syntactic or semantic analysis of the text; the results are fast but only heuristic with regards to actual semantics of the text. Questions?:  Questions?

Related presentations


Other presentations created by Barbara

Solar System
17. 06. 2007
0 views

Solar System

Advanced SQL Injection
30. 08. 2007
0 views

Advanced SQL Injection

PrivateExchange
22. 04. 2008
0 views

PrivateExchange

07 fordjob1
17. 04. 2008
0 views

07 fordjob1

20061011114434853
13. 04. 2008
0 views

20061011114434853

Bruce Lambert Army Corps
10. 04. 2008
0 views

Bruce Lambert Army Corps

SPAC2007 Juan Rodriguez
09. 04. 2008
0 views

SPAC2007 Juan Rodriguez

Chapter7
07. 04. 2008
0 views

Chapter7

tourism chapter 04
30. 03. 2008
0 views

tourism chapter 04

LAC International Trade
28. 03. 2008
0 views

LAC International Trade

feb2006final
27. 03. 2008
0 views

feb2006final

virtualcommunities
26. 03. 2008
0 views

virtualcommunities

Mickey Mouse
26. 06. 2007
0 views

Mickey Mouse

1who gets tb in nyc
27. 09. 2007
0 views

1who gets tb in nyc

lijian
12. 10. 2007
0 views

lijian

O2 Diesel
08. 11. 2007
0 views

O2 Diesel

American Romanticism
30. 08. 2007
0 views

American Romanticism

233nm60
30. 08. 2007
0 views

233nm60

MBA Lecture Series v2
30. 08. 2007
0 views

MBA Lecture Series v2

hep2005 talk MarkVagins
09. 10. 2007
0 views

hep2005 talk MarkVagins

Control Tech
05. 12. 2007
0 views

Control Tech

DasuCMSTriggerUCSD
07. 10. 2007
0 views

DasuCMSTriggerUCSD

ams ppt
30. 08. 2007
0 views

ams ppt

Question Answering
16. 11. 2007
0 views

Question Answering

Facts x about Finland
22. 11. 2007
0 views

Facts x about Finland

OWAS PAppSecEU2006 CLASP Project
30. 08. 2007
0 views

OWAS PAppSecEU2006 CLASP Project

OWASP Flyer Sep06
30. 08. 2007
0 views

OWASP Flyer Sep06

fun with hyperplanes 2007
28. 12. 2007
0 views

fun with hyperplanes 2007

american history
28. 12. 2007
0 views

american history

Frank Garber Presentation
02. 01. 2008
0 views

Frank Garber Presentation

DPS07 65 01 Fritzius
03. 01. 2008
0 views

DPS07 65 01 Fritzius

Teaching Political Sociology
04. 01. 2008
0 views

Teaching Political Sociology

Gaming in Education
07. 01. 2008
0 views

Gaming in Education

Plume tracking hardware
07. 01. 2008
0 views

Plume tracking hardware

Altera
28. 11. 2007
0 views

Altera

dead reckon cdr
07. 01. 2008
0 views

dead reckon cdr

Infections 3
04. 12. 2007
0 views

Infections 3

CMC IR1001
27. 09. 2007
0 views

CMC IR1001

class2 3
16. 11. 2007
0 views

class2 3

mixload
06. 11. 2007
0 views

mixload

web query 0609
07. 11. 2007
0 views

web query 0609

FSA
27. 12. 2007
0 views

FSA

CompanyDossier
29. 09. 2007
0 views

CompanyDossier

Hunting For Black Holes
28. 11. 2007
0 views

Hunting For Black Holes

DAR
20. 02. 2008
0 views

DAR

8 Soci 1015 Chapter7 Family
24. 02. 2008
0 views

8 Soci 1015 Chapter7 Family

ABSSEI Oswald
29. 02. 2008
0 views

ABSSEI Oswald

NeMO Curr Part3 v2
26. 06. 2007
0 views

NeMO Curr Part3 v2

nelson sheinberg Presentation
26. 06. 2007
0 views

nelson sheinberg Presentation

n0002 SPIE1
26. 06. 2007
0 views

n0002 SPIE1

Metric System 1
26. 06. 2007
0 views

Metric System 1

media kit
26. 06. 2007
0 views

media kit

March 14 PMI Presentation
26. 06. 2007
0 views

March 14 PMI Presentation

fountain of age
26. 06. 2007
0 views

fountain of age

Lifting Equation
13. 12. 2007
0 views

Lifting Equation

Dietary Guidelines
04. 03. 2008
0 views

Dietary Guidelines

upshur pc1
10. 03. 2008
0 views

upshur pc1

crossref
30. 08. 2007
0 views

crossref

ddbppt
20. 11. 2007
0 views

ddbppt

DEPBasicsCourse
30. 12. 2007
0 views

DEPBasicsCourse

guerra
12. 11. 2007
0 views

guerra

James F Cooper
30. 08. 2007
0 views

James F Cooper

lubin talk
03. 01. 2008
0 views

lubin talk

NDD presentation compressed
30. 08. 2007
0 views

NDD presentation compressed

madcooper
07. 12. 2007
0 views

madcooper

graduacion1
01. 01. 2008
0 views

graduacion1

GBIF demo Japan081003
27. 11. 2007
0 views

GBIF demo Japan081003

20061019 1732 oberauer hql06
15. 11. 2007
0 views

20061019 1732 oberauer hql06

phpulse oct
05. 01. 2008
0 views

phpulse oct

media searching
26. 06. 2007
0 views

media searching

Smith Core values
17. 06. 2007
0 views

Smith Core values

Smith1
17. 06. 2007
0 views

Smith1

Significance of the Cross
17. 06. 2007
0 views

Significance of the Cross

Sharp
17. 06. 2007
0 views

Sharp

section 2 attitude to food
17. 06. 2007
0 views

section 2 attitude to food

Section4 5
17. 06. 2007
0 views

Section4 5

Spirituality
17. 06. 2007
0 views

Spirituality

sonnet presentation
17. 06. 2007
0 views

sonnet presentation

Star addition tutorial
17. 06. 2007
0 views

Star addition tutorial

stand up comedy
17. 06. 2007
0 views

stand up comedy

SS 1SBrown
17. 06. 2007
0 views

SS 1SBrown

Emerson Transcendentalism
30. 08. 2007
0 views

Emerson Transcendentalism

ABinEurope
23. 11. 2007
0 views

ABinEurope

oct04ach
05. 11. 2007
0 views

oct04ach

SCP2
17. 06. 2007
0 views

SCP2

transcendentalism
30. 08. 2007
0 views

transcendentalism

micro ch03 presentation
04. 10. 2007
0 views

micro ch03 presentation

SC morning
17. 06. 2007
0 views

SC morning

ISIC cobrandNEUenglish
18. 03. 2008
0 views

ISIC cobrandNEUenglish

02b LisbonWeb
30. 12. 2007
0 views

02b LisbonWeb

ProvenceArchitecture
05. 11. 2007
0 views

ProvenceArchitecture

san diego 04
01. 11. 2007
0 views

san diego 04

noemie 2
26. 06. 2007
0 views

noemie 2

Community Service PP 06 FOR WEB
05. 11. 2007
0 views

Community Service PP 06 FOR WEB

Sections3 7
17. 06. 2007
0 views

Sections3 7

ECE TRANS WP29 GRSP 41 inf09e
26. 11. 2007
0 views

ECE TRANS WP29 GRSP 41 inf09e

srwg graz
26. 11. 2007
0 views

srwg graz

Meydan
23. 11. 2007
0 views

Meydan

LWS05
02. 11. 2007
0 views

LWS05

mal 2005 bra
30. 08. 2007
0 views

mal 2005 bra

Standards Aligned Classroom
17. 06. 2007
0 views

Standards Aligned Classroom

steenkampNVDRS
06. 03. 2008
0 views

steenkampNVDRS