lecture16

Information about lecture16

Published on September 6, 2007

Author: Waldarrama

Source: authorstream.com

Content

SIMS 290-2: Applied Natural Language Processing:  SIMS 290-2: Applied Natural Language Processing Marti Hearst October 30, 2006 Some slides by Preslav Nakov and Eibe Frank     Today:  Today The Newsgroups Text Collection WEKA: Exporer WEKA: Experimenter Python Interface to WEKA Slide3:  Source: originally collected by Ken Lang Content and structure: approximately 20,000 newsgroup documents 19,997 originally 18,828 without duplicates partitioned evenly across 20 different newsgroups we are only using a subset (6 newsgroups) Some categories are strongly related (and thus hard to discriminate): 20 Newsgroups Data Set http://people.csail.mit.edu/u/j/jrennie/public_html/20Newsgroups/ computers Slide4:  Sample Posting: 'talk.politics.guns' From: [email protected] (C. D. Tavares) Subject: Re: Congress to review ATF's status In article andlt;[email protected];, [email protected] (Larry Cipriani) writes: andgt; WASHINGTON (UPI) -- As part of its investigation of the deadly andgt; confrontation with a Texas cult, Congress will consider whether the andgt; Bureau of Alcohol, Tobacco and Firearms should be moved from the andgt; Treasury Department to the Justice Department, senators said Wednesday. andgt; The idea will be considered because of the violent and fatal events andgt; at the beginning and end of the agency's confrontation with the Branch andgt; Davidian cult. Of course. When the catbox begines to smell, simply transfer its contents into the potted plant in the foyer. 'Why Hillary! Your government smells so... FRESH!' -- [email protected] --If you believe that I speak for my company, OR [email protected] write today for my special Investors' Packet... reply from subject signature Need special handling during feature extraction… … writes: Slide5:  The 20 Newsgroups Text Collection WEKA: Exporer WEKA: Experimenter Python Interface to WEKA WEKA: Real-time Demo Slide6:  WEKA: The Bird Copyright: Martin Kramer ([email protected]), University of Waikato, New Zealand WEKA: the software:  WEKA: the software Waikato Environment for Knowledge Analysis Collection of state-of-the-art machine learning algorithms and data processing tools implemented in Java Released under the GPL Support for the whole process of experimental data mining Preparation of input data Statistical evaluation of learning schemes Visualization of input data and the result of learning Used for education, research and applications Complements 'Data Mining' by Witten andamp; Frank Main Features:  Main Features 49 data preprocessing tools 76 classification/regression algorithms 8 clustering algorithms 15 attribute/subset evaluators + 10 search algorithms for feature selection 3 algorithms for finding association rules 3 graphical user interfaces 'The Explorer' (exploratory data analysis) 'The Experimenter' (experimental environment) 'The KnowledgeFlow' (new process model inspired interface) Projects based on WEKA:  Projects based on WEKA Incorporate/wrap WEKA GRB Tool Shed - a tool to aid gamma ray burst research YALE - facility for large scale ML experiments GATE - NLP workbench with a WEKA interface Judge - document clustering and classification Extend/modify WEKA BioWeka - extension library for knowledge discovery in biology WekaMetal - meta learning extension to WEKA Weka-Parallel - parallel processing for WEKA Grid Weka - grid computing using WEKA Weka-CG - computational genetics tool library The WEKA Project Today (2006):  The WEKA Project Today (2006) Funding for the next two years Goal of the project remains the same People 6 staff 2 postdocs 3 PhD students 3 MSc students 2 research programmers Slide11:  WEKA: The Software Toolkit Machine learning/data mining software in Java GNU License Used for research, education and applications Complements 'Data Mining' by Witten andamp; Frank Main features: data pre-processing tools learning algorithms evaluation methods graphical interface (incl. data visualization) environment for comparing learning algorithms http://www.cs.waikato.ac.nz/ml/weka http://sourceforge.net/projects/weka/ Slide12:  WEKA: Terminology Some synonyms/explanations for the terms used by WEKA, which may differ from what we use: Attribute: feature Relation: collection of examples Instance: collection in use Class: category Slide13:  WEKA GUI Chooser java -Xmx1000M -jar weka.jar Our Toy Example:  Our Toy Example We demonstrate WEKA on a simple example: 3 categories from 'Newsgroups': misc.forsale, rec.sport.hockey, comp.graphics 20 documents per category features: words converted to lowercase frequency 2 or more required stopwords removed Explorer: Pre-Processing The Data:  Explorer: Pre-Processing The Data WEKA can import data from: files: ARFF, CSV, C4.5, binary URL SQL database (using JDBC) Pre-processing tools (filters) are used for: Discretization, normalization, resampling, attribute selection, transforming and combining attributes, etc. Slide16:  List of attributes (last: class variable) Frequency and categories for the selected attribute Statistics about the values of the selected attribute Classification Filter selection Manual attribute selection Statistical attribute selection Preprocessing The Preprocessing Tab Explorer: Building “Classifiers”:  Explorer: Building 'Classifiers' Classifiers in WEKA are models for: classification (predict a nominal class) regression (predict a numerical quantity) Learning algorithms: Naïve Bayes, decision trees, kNN, support vector machines, multi-layer perceptron, logistic regression, etc. Meta-classifiers: cannot be used alone always combined with a learning algorithm examples: boosting, bagging etc. Slide18:  Choice of classifier The attribute whose value is to be predicted from the values of the remaining ones. Default is the last attribute. Here (in our toy example) it is named 'class'. Cross-validation: split the data into e.g. 10 folds and 10 times train on 9 folds and test on the remaining one The Classification Tab Slide19:  Choosing a classifier Slide20:  Slide21:  False: Gaussian True: kernels (better) displays synopsis and options numerical to nominal conversion by discretization outputs additional information Slide22:  Slide23:  Slide24:  all other numbers can be obtained from it different/easy class accuracy Slide25:  Contains information about the actual and the predicted classification All measures can be derived from it: accuracy: (a+d)/(a+b+c+d) recall: d/(c+d) =andgt; R precision: d/(b+d) =andgt; P F-measure: 2PR/(P+R) false positive (FP) rate: b/(a+b) true negative (TN) rate: a/(a+b) false negative (FN) rate: c/(c+d) These extend for more than 2 classes: see previous lecture slides for details Confusion matrix Slide26:  Outputs the probability distribution for each example Predictions Output Slide27:  Probability distribution for a wrong example: predicted 1 instead of 3 Naïve Bayes makes incorrect conditional independence assumptions and typically is over-confident in its prediction regardless of whether it is correct or not. Predictions Output Slide28:  Error Visualization Slide29:  Error Visualization Little squares designate errors Axes show example number Slide30:  Running on Test Set Slide31:  Find which attributes are the most predictive ones Two parts: search method: best-first, forward selection, random, exhaustive, genetic algorithm, ranking evaluation method: information gain, chi-squared, etc. Very flexible: WEKA allows (almost) arbitrary combinations of these two Explorer: Attribute Selection Slide32:  Individual Features Ranking Slide33:  misc.forsale comp.graphics rec.sport.hockey Individual Features Ranking Slide34:  misc.forsale comp.graphics rec.sport.hockey ??? random number seed Individual Features Ranking Feature Interactions (Advanced):  Feature Interactions (Advanced) C B A category feature feature importance of feature B importance of feature A Feature Interactions (Advanced):  Feature Interactions (Advanced) C B A category feature feature importance of feature B importance of feature A Feature Subsets Selection:  Feature Subsets Selection Problem illustration Full set Empty set Enumeration Search Exhaustive/Complete (enumeration/branchandamp;bounding) Heuristic (sequential forward/backward) Stochastic (generate/evaluate) Individual features or subsets generation/evaluation Slide38:  Features Subsets Selection Slide39:  misc.forsale comp.graphics rec.sport.hockey 17,309 subsets considered 21 attributes selected Features Subsets Selection Slide40:  Saving the Selected Features All we can do from this tab is to save the buffer in a text file. Not very useful... But we can also perform feature selection during the pre-processing step... (the following slides) Slide41:  Features Selection on Preprocessing Slide42:  Features Selection on Preprocessing Slide43:  Features Selection on Preprocessing 679 attributes: 678 + 1 (for the class) Slide44:  Features Selection on Preprocessing Just 22 attributes remain: 21 + 1 (for the class) Slide45:  Run Naïve Bayes With the 21 Features higher accuracy 21 Attributes Slide46:  different/easy class accuracy (AGAIN) Naïve Bayes With All Features ALL 679 Attributes (repeated slide) Slide47:  WEKA has weird naming for some algorithms Here are some translations: Naïve Bayes: weka.classifiers.bayes.NaiveBayes Perceptron: weka.classifiers.functions.VotedPerceptron Decision tree: weka.classifiers.trees.J48 Support vector machines: weka.classifiers.functions.SMO k nearest neighbor: weka.classifiers.lazy.IBk Some of these are more sophisticated versions of the classic algorithms e.g. the classic Naïve Bayes seems to be missing A good alternative is the Multinomial Naïve Bayes model Some Important Algorithms Slide48:  The 20 Newsgroups Text Collection WEKA: Explorer WEKA: Experimenter Python Interface to WEKA WEKA: Real-time Demo Slide49:  Experimenter makes it easy to compare the performance of different learning schemes Problems: classification regression Results: written into file or database Evaluation options: cross-validation learning curve hold-out Can also iterate over different parameter settings Significance-testing built in! Performing Experiments Slide50:  Experiments Setup Slide51:  Experiments Setup Slide52:  Experiments Setup CSV file: can be open in Excel datasets algorithms Slide53:  Experiments Setup Slide54:  Experiments Setup Slide55:  Experiments Setup Slide56:  Experiments Setup Slide57:  Experiments Setup accuracy SVM is the best Decision tree is the worst SVM is statistically better than Naïve Bayes Decision tree is statistically worse than Naïve Bayes Slide58:  Experiments: Excel Results are output into an CSV file, which can be read in Excel! Slide59:  The Newsgroups Text Collection WEKA: Explorer WEKA: Experimenter Python Interface to WEKA Slide60:  @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present ... WEKA File Format: ARFF Other attribute types: String Date Numerical attribute Nominal attribute Missing value Slide61:  Value 0 is not represented explicitly Same header (i.e @relation and @attribute tags) the @data section is different Instead of @data 0, X, 0, Y, 'class A' 0, 0, W, 0, 'class B' We have @data {1 X, 3 Y, 4 'class A'} {2 W, 4 'class B'} This is especially useful for textual data (why?) WEKA File Format: Sparse ARFF Slide62:  Python Interface to WEKA This is just to get you started Assumes the newsgroups collection Extracts simple features currently just single word features Uses a simple tokenizer which removes punctuation uses a stoplist lowercases the words Includes filtering code currently eliminates numbers Features are weighted by frequency within document Produces a sparse ARFF file to be used by WEKA Slide63:  Python Interface to WEKA Allows you to specify: Which directory to read files from which newsgroups to use the number of documents for training each newsgroup the number of features to retain Slide64:  Python Interface to WEKA Things to (optionally) add or change: an option to not use stopwords an option to retain capitalization regular expression pattern a feature should match other non-word-based features morphological normalization a minimum threshold for the number of time a term occurs before it can be counted as a feature tf.idf weighting on terms your idea goes here Slide65:  Python Interface to WEKA TF.IDF: tij  log(N/ni) TF tij: frequency of term i in document j this is how features are currently weighted IDF: log(N/ni) ni: number of documents containing term i N: total number of documents Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code Python Weka Code:  Python Weka Code ARFF file:  ARFF file … ARFF file:  ARFF file … Assignment:  Assignment Due November 13. Work individually on this one Objective is to use the training set to get the best features and learning model you can. FREEZE this. Then run one time only on the test set. This is a realistic way to see how well your algorithm does on unseen data. Next Time:  Next Time Machine learning algorithms

Related presentations


Other presentations created by Waldarrama

ISP 20071031
30. 11. 2007
0 views

ISP 20071031

Mexican Revolution
13. 04. 2008
0 views

Mexican Revolution

CHINA
26. 03. 2008
0 views

CHINA

God is Love
17. 06. 2007
0 views

God is Love

GENOCIDE FRAMEWORK
28. 12. 2007
0 views

GENOCIDE FRAMEWORK

quidnunc
22. 04. 2008
0 views

quidnunc

berlino
17. 04. 2008
0 views

berlino

pandemics
10. 04. 2008
0 views

pandemics

WF Surface Water
07. 04. 2008
0 views

WF Surface Water

LopezBGET12May05
30. 03. 2008
0 views

LopezBGET12May05

Tut Prager
28. 03. 2008
0 views

Tut Prager

AMY IMS CLIVARSSG15
27. 03. 2008
0 views

AMY IMS CLIVARSSG15

boone
04. 10. 2007
0 views

boone

After The Tornado
05. 10. 2007
0 views

After The Tornado

Tornadoes
07. 10. 2007
0 views

Tornadoes

gp 9 forest resources
10. 10. 2007
0 views

gp 9 forest resources

maki
06. 09. 2007
0 views

maki

Teaching Hockey Sense
06. 09. 2007
0 views

Teaching Hockey Sense

Mouth Protection Info For Clinic
06. 09. 2007
0 views

Mouth Protection Info For Clinic

peter huybers
06. 09. 2007
0 views

peter huybers

P E at Arnold House
06. 09. 2007
0 views

P E at Arnold House

The Kerr Metric
29. 11. 2007
0 views

The Kerr Metric

BHEU2004 NF SP EWS v11
03. 12. 2007
0 views

BHEU2004 NF SP EWS v11

Presentation Bejakovic
05. 12. 2007
0 views

Presentation Bejakovic

911
02. 11. 2007
0 views

911

dalhousie
12. 11. 2007
0 views

dalhousie

SEGUNDA GUERRA MUNDIAL
13. 11. 2007
0 views

SEGUNDA GUERRA MUNDIAL

Svarc
14. 11. 2007
0 views

Svarc

7 CECCHINI Marco
16. 11. 2007
0 views

7 CECCHINI Marco

Las Mujeres Jaguar
20. 11. 2007
0 views

Las Mujeres Jaguar

visual studio 2008 linq
28. 11. 2007
0 views

visual studio 2008 linq

WoodandRyan
24. 12. 2007
0 views

WoodandRyan

CreditCards
25. 12. 2007
0 views

CreditCards

southeastasia
28. 12. 2007
0 views

southeastasia

BC8 11 2
01. 01. 2008
0 views

BC8 11 2

dw space
02. 01. 2008
0 views

dw space

NHSL Dont forget workforce
07. 01. 2008
0 views

NHSL Dont forget workforce

Prasser Project Conf 2006 final
26. 11. 2007
0 views

Prasser Project Conf 2006 final

ASD BASIC
28. 12. 2007
0 views

ASD BASIC

OQ Presentation2
04. 12. 2007
0 views

OQ Presentation2

HIPAA Education BasicFinal0103
23. 12. 2007
0 views

HIPAA Education BasicFinal0103

NCIA CARDÃ ACA
28. 12. 2007
0 views

NCIA CARDÃ ACA

2002 annual meeting print
19. 02. 2008
0 views

2002 annual meeting print

Gang Primer 1ID1
26. 02. 2008
0 views

Gang Primer 1ID1

Prasad p3
06. 09. 2007
0 views

Prasad p3

PotatoesI
04. 03. 2008
0 views

PotatoesI

UNIT ONE INTRODUCTION
06. 03. 2008
0 views

UNIT ONE INTRODUCTION

Prezentacja4
18. 03. 2008
0 views

Prezentacja4

NC2005 Denbeck
06. 09. 2007
0 views

NC2005 Denbeck

Ceremonial Speech
31. 12. 2007
0 views

Ceremonial Speech

03conv d3 champs
06. 09. 2007
0 views

03conv d3 champs

BCH Slide PresentationFinal
06. 09. 2007
0 views

BCH Slide PresentationFinal

G050504 00
03. 10. 2007
0 views

G050504 00

02 TCSS Semi Formal Dance
27. 11. 2007
0 views

02 TCSS Semi Formal Dance

Changes in Medina County
11. 12. 2007
0 views

Changes in Medina County

sunum39
21. 11. 2007
0 views

sunum39

S14 44
17. 12. 2007
0 views

S14 44

God Embraces vs 2
17. 06. 2007
0 views

God Embraces vs 2

gender issues
17. 06. 2007
0 views

gender issues

Funny Bunny Camille Page
17. 06. 2007
0 views

Funny Bunny Camille Page

Funny Turns
17. 06. 2007
0 views

Funny Turns

history of comedy presentation
17. 06. 2007
0 views

history of comedy presentation

history of english
17. 06. 2007
0 views

history of english

hinman romeo
17. 06. 2007
0 views

hinman romeo

Helping Children Love to Read
17. 06. 2007
0 views

Helping Children Love to Read

health humor
17. 06. 2007
0 views

health humor

happiness Belarus
17. 06. 2007
0 views

happiness Belarus

group F
17. 06. 2007
0 views

group F

Gripping
17. 06. 2007
0 views

Gripping

gallows humor
17. 06. 2007
0 views

gallows humor

fwe 05
17. 06. 2007
0 views

fwe 05

bremsstrahlung nss2006
22. 11. 2007
0 views

bremsstrahlung nss2006

praes lehnert
14. 03. 2008
0 views

praes lehnert

Dabholkar2
28. 11. 2007
0 views

Dabholkar2

HELP PP
17. 06. 2007
0 views

HELP PP

Gaubatz
06. 11. 2007
0 views

Gaubatz

Women Energy
04. 01. 2008
0 views

Women Energy

chi course06 4 23
03. 10. 2007
0 views

chi course06 4 23

ALDA
06. 09. 2007
0 views

ALDA

CobraKai MarketingPresentation
06. 09. 2007
0 views

CobraKai MarketingPresentation

NAU Hockey Club
06. 09. 2007
0 views

NAU Hockey Club

AGM CoachingandNewNCCP
06. 09. 2007
0 views

AGM CoachingandNewNCCP

hackman CLC
17. 06. 2007
0 views

hackman CLC

Yi Capstone071604
07. 12. 2007
0 views

Yi Capstone071604

Hotchkiss REU03
07. 11. 2007
0 views

Hotchkiss REU03

CASE ITINERARIO
01. 11. 2007
0 views

CASE ITINERARIO