bwright

Information about bwright

Published on January 16, 2008

Author: Panfilo

Source: authorstream.com

Content

How to convince your friends NOT to misuse raw scores:  How to convince your friends NOT to misuse raw scores Benjamin D. Wright Institute for Objective Measurement & MESA Psychometric Laboratory [email protected] THE TROUBLE WITH RAW SCORES:  THE TROUBLE WITH RAW SCORES The BROKEN BUCKET of Missing Data The CRUMBLING CATEGORIES of Lumpy Ratings The DIRTY DATA of Unpredictable Responses The PERVERSE PRECISION of Extreme Scores The RUBBER RULER of Irregular Intervals and Squashed Extremes The BROKEN BUCKET of Missing Data :  The BROKEN BUCKET of Missing Data Compare Two Patients on an 8-item, 7-category Functional Independence Measure: Patient A: 2 3 3 m m m m 4 = 12 Patient B: 1 2 2 2 2 2 2 3 = 16 Which Patient is more Able? Patient B has the higher score 16 > 12 on all 8 items, BUT On the 4 Items A and B have in common Patient A: 2 3 3 m m m m 4 = 12 Patient B: 1 2 2 m m m m 3 = 8 Patient A has the higher score of 12 > 8 Are you sure you want to misuse missing-data-leaking raw scores for missing-data-impervious measures? The CRUMBLING CATEGORIES of Lumpy Ratings:  The CRUMBLING CATEGORIES of Lumpy Ratings Rating Forms Offer Equally Spaced Categories 1 . 2 . 3 . 4 . 5 . 6 . 7 But Raters Reply with Unequally Spaced Responses 1 . . . 2 . . 3 4 . 5 . . 6 . . . . . 7 The Measure Distance of One More Point from Category 1 to 2 can be FOUR times BIGGER than The Measure Distance of One More Point from Category 3 to 4 !! Are you sure you want to mistake Lumpy Ratings for Equal Interval Measures? The DIRTY DATA of Unpredictable Responses:  The DIRTY DATA of Unpredictable Responses When Item Responses are Arranged from Easy Items to Hard Items you Expect Response Patterns like: 7 7 6 6 6 5 5 4 = 46 and 4 4 3 3 2 1 1 1 = 19 BUT suppose you get: 7 7 6 6 {1}5 5 4 = 41 ? or 4 4 3 3 {7}1 1 1 = 24 ? What then? Raw Scores are Blind to Unpredictable Responses. Only Quality Control of Well-Constructed Measures Tells you about Response Surprises Are you sure you want to suffer raw-score-dirty-data blindness instead of enjoying data-vigilant measures? The PERVERSE PRECISION of Extreme Scores:  The PERVERSE PRECISION of Extreme Scores The Statistical Precision of a Raw Score is MAXIMUM at exactly the place where the Information a Raw Score Provides is MINIMUM When a person gets the lowest possible score, their raw score precision is perfect. We know exactly the score their low ability implies. But we have no idea how far Below that Score their ability might be! It is the same with the highest possible score. We know exactly the score their high ability implies. But we have no idea how far Above that Score their ability might be! They are Off-Our-Scale and Our Precision for their Unknown Measure is ZERO! Are you sure you want to mistake imprecise raw scores for precise measures? The RUBBER RULER of Irregular Intervals and Squashed Extremes:  The RUBBER RULER of Irregular Intervals and Squashed Extremes When our items bunch in clumps of equally difficult items then a count of one more right answer within a clump implies only a little increase in our ability. But when we leap ahead and our next right answer is in a distinctly harder clump, then we see that this one more right implies a large increase in our ability. As for the ends of the test where one more right is from 0 to 1 or from all but one to all. Then the implied change in our ability is infinite. WHAT ARE VARIABLES?:  WHAT ARE VARIABLES? Length and weight may be real variables. But we construct their units of measure. Inches and ounces are our creations - Our own imaginative constructions. A variable is an amount of something which we can always picture as a distance >From Less -------------------------> To More We can arrange to experience evidence of this "something". But its measurement line and its units of measurement are up to us to construct. EVIDENCE OF A VARIABLE?:  EVIDENCE OF A VARIABLE? The variable and its evidence could be: length benchmarks exceeded health symptoms absent ability problems solved skill tasks completed attitude assertions condoned We can arrange to provoke occurrences of evidence and count how many pieces occur. But these counts are not measures. REQUIREMENTS FOR MEASUREMENT:  REQUIREMENTS FOR MEASUREMENT Pieces of evidence must be concrete to be observed. This necessary reality keeps them uneven in size. To measure we need an even abstraction, a line marked out in abstractly equal units. Pieces of evidence are unstable. They appear and disappear by accident. They are only probable signs of the variable which they are designed to manifest. To measure, we must find a way to connect the pieces of evidence we can arrange to observe to the probabilities of the measures we want. WHAT IS MEASUREMENT?:  WHAT IS MEASUREMENT? DISTANCE (Length) was our First Variable COUNTING Steps and Fingers was our First Measuring Operation The Trouble with Counting is its UNEQUAL UNITS How many apples fill a basket? How many oranges squeeze a glass? You may not believe it. But can we mix apples and oranges? We do it all of the time, by WEIGHING them! CONSTRUCTING MEASURES:  CONSTRUCTING MEASURES But weighing is a constructed abstraction.There are no tangible equal units. We have to invent them. Equal feet are abstracted from real feet. Equal pounds are abstract real weights We construct our instrumentation of the variables: length and weight to approximate units equal enough to serve our practical purposes We measure so that we can use the past to plan and navigate the future. But the future is by definition UNCERTAIN HANDLING UNCERTAINTY:  HANDLING UNCERTAINTY Imagine two batters: Smith bats 400 and Jones bats 200 So which one will hit at their next batter-up? No way to know ahead of time. Even Smith has only a 4 out of 10 record. We can’t wait to find out. So which one shall we send to the plate? Smith's odds for a hit are 2/3; Jones' odds for a hit are only 1/4 Smith odds for a hit are 8/3 times better than Jones'. Even though we know nothing for sure, Does any doubt remain as to who to send to bat? That's how we handle uncertainty. We use past experience to estimate PROBABILITIES and use these probabilities to forsee the future. COUNTING ABSTRACT UNITS:  COUNTING ABSTRACT UNITS To finish this job we have to construct a reproducible transition from counting concrete events, like right answers, observed or reported symptoms, relative agreements, frequency or importance categories to counting abstract units of equal size and wide generality. How can we do this? INVERSE PROBABILITY:  INVERSE PROBABILITY To deal with the uncertainty we ask Bernoulli, Bayes and Laplace and interpret our observation X as evidence of its occurence probability Px To construct unit equality we ask Campbell, Thurstone, Rasch and Luce & Tukey and define Px to satisfy the equation: log[Pnix/(1-Pnix)] = Bn - Di Pnix is the probability of a successful response Xni being produced by person n to item i Bn is the ability of person n Di is the difficulty of item I The construction of equal size and hence additive units is called CONJOINT ADDITIVITY Slide17:  For situations where Xni occurs in incremental steps such as Xni = 0,1,2,3,,,M This simple solution generalizes to log[Pnix/Pnix-1] = Bn - Di - Fix For situations where Xnijk = 0,M occurs as the result of a Rater j rating the performance of a Person n on a Task k This MEASUREMENT MODEL becomes log[Pnijkx/Pnijkx-1] = Bn - Di - Cj - Ak - Fix CONJOINT ADDITIVITY CLOSING THE DEAL:  CLOSING THE DEAL Raw scores cause problems with: missing data lumpy ratings unpredictable responses extreme scores irregular intervals and squashed extremes Rasch measurement provides a solution to these problems by: abstracting units of measurement using probabilities to predict futures constructing equal-sized intervals

Related presentations


Other presentations created by Panfilo

Fine Motor Olympics
21. 04. 2008
0 views

Fine Motor Olympics

S04 HRM Change Management
19. 01. 2008
0 views

S04 HRM Change Management

multimedia01
16. 03. 2008
0 views

multimedia01

OLYMPIC GAMES
02. 05. 2008
0 views

OLYMPIC GAMES

Apple Harvest
08. 01. 2008
0 views

Apple Harvest

Safety Tips Recreation
10. 01. 2008
0 views

Safety Tips Recreation

BS Evaluation
10. 01. 2008
0 views

BS Evaluation

rendering
11. 01. 2008
0 views

rendering

Transitional Society
12. 01. 2008
0 views

Transitional Society

Jeopardy game
14. 01. 2008
0 views

Jeopardy game

Classical Conditioning Review
15. 01. 2008
0 views

Classical Conditioning Review

interwar 7oct
16. 01. 2008
0 views

interwar 7oct

DDCI1 1
16. 01. 2008
0 views

DDCI1 1

College English Book 4 Unit 7
16. 01. 2008
0 views

College English Book 4 Unit 7

press
17. 01. 2008
0 views

press

Introduction lesson
17. 01. 2008
0 views

Introduction lesson

poison happenings
18. 01. 2008
0 views

poison happenings

2FA
20. 01. 2008
0 views

2FA

newspaper society ppt
21. 01. 2008
0 views

newspaper society ppt

river crossing
04. 02. 2008
0 views

river crossing

India Arranged Marriage
05. 02. 2008
0 views

India Arranged Marriage

AVT Ideas for 2006
26. 01. 2008
0 views

AVT Ideas for 2006

Catholic ActionShow111105
29. 01. 2008
0 views

Catholic ActionShow111105

world
14. 02. 2008
0 views

world

FaPro 01
20. 02. 2008
0 views

FaPro 01

Britney Spears
27. 02. 2008
0 views

Britney Spears

BE
28. 01. 2008
0 views

BE

office ergonomics
05. 03. 2008
0 views

office ergonomics

WhatWorksWhatDoesntS EPT06
04. 02. 2008
0 views

WhatWorksWhatDoesntS EPT06

McClelland
10. 03. 2008
0 views

McClelland

price discrimination
19. 03. 2008
0 views

price discrimination

DavisAtHIIT
21. 03. 2008
0 views

DavisAtHIIT

java a to z
22. 03. 2008
0 views

java a to z

LIUtalk
24. 01. 2008
0 views

LIUtalk

The Ancient Olympics
14. 04. 2008
0 views

The Ancient Olympics

chapt11LectureOutlin es
17. 04. 2008
0 views

chapt11LectureOutlin es

Hannes de Bruin
29. 01. 2008
0 views

Hannes de Bruin

Marilyn Hutchinson PP
22. 04. 2008
0 views

Marilyn Hutchinson PP

B1 U01 A1
24. 04. 2008
0 views

B1 U01 A1

KSubraminam
07. 05. 2008
0 views

KSubraminam

Dave Juday Mvd Sept 30
08. 05. 2008
0 views

Dave Juday Mvd Sept 30

abrikosov
30. 04. 2008
0 views

abrikosov

ASTMCIC2002
23. 01. 2008
0 views

ASTMCIC2002

difffeatures
02. 05. 2008
0 views

difffeatures

Monica Candal Presentation
06. 02. 2008
0 views

Monica Candal Presentation

10 35 Overview of IHY
22. 01. 2008
0 views

10 35 Overview of IHY

erica jobson workshop
28. 01. 2008
0 views

erica jobson workshop

20071126 wyd bilingue
05. 02. 2008
0 views

20071126 wyd bilingue

Martian Cement
11. 02. 2008
0 views

Martian Cement

RR Awards 04 Nominees Winners
12. 01. 2008
0 views

RR Awards 04 Nominees Winners

Zhou REGE
30. 01. 2008
0 views

Zhou REGE

0202 agu kerfoot
25. 02. 2008
0 views

0202 agu kerfoot

Loughman
20. 02. 2008
0 views

Loughman

shaver flight
15. 01. 2008
0 views

shaver flight

Koket
07. 02. 2008
0 views

Koket