NRCCWDT DataAbuse

Information about NRCCWDT DataAbuse

Published on January 13, 2008

Author: Susett

Source: authorstream.com

Content

Data Use vs. Misuse: The challenging nature of publicly available data :  Data Use vs. Misuse: The challenging nature of publicly available data Emily Putnam-Hornstein, MSW Barbara Needell, MSW, PhD Center for Social Services Research University of California at Berkeley National Child Welfare Data and Technology Conference July 20, 2007 The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation. (a.k.a., “Numbers Gone Wild”) Overview…:  Overview… Child Welfare Data in California: History The Good Stuff: Public Website The Bad Stuff: (Real) Examples of Data Misuse Strategies for educating users / discussion California’s Child Welfare Data:  California’s Child Welfare Data State supervised, county administered child welfare system 58 Diverse Counties Longstanding Interagency Agreement Quarterly Data Reports to State and County officials Funding from CDSS and Stuart Foundation Data Publicly Available… http://cssr.berkeley.edu/ucb_childwelfare/:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide5:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide6:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide7:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide8:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide9:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide10:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide11:  http://cssr.berkeley.edu/ucb_childwelfare/ Slide12:  http://cssr.berkeley.edu/ucb_childwelfare/ Public Data: Putting it all out there:  Public Data: Putting it all out there PROS: Greater performance accountability Community awareness and involvement, encourages public-private partnerships Ability to track improvement over time, identify areas where programmatic adjustments are needed County/County and County/State collaboration Transparency Dialogue Public Data: Putting it all out there:  CONS: Potential for misuse, misinterpretation, and misrepresentation Available to those with agendas or looking to create a sensational headline Misunderstood data can lead to the wrong policy decisions “Torture numbers, and they’ll confess to anything” (Gregg Easterbrook) Public Data: Putting it all out there Slide15:  There are three kinds of lies: Lies, Damned Lies and Statistics Misused Statistics ^ Six Ways to Misuse Data (without actually lying!)::  Six Ways to Misuse Data (without actually lying!): Rank Data Compare Apples and Oranges Use ‘snapshots’ of Small Samples Rely on Unrepresentative Findings Logically ‘flip’ Statistics Falsely Assume an Association to be Causal 1) Rank Data:  1) Rank Data Two streets in Anytown, CA…. It’s all relative… And SOMEONE will always be ranked last (and first) Poverty Blvd $$ Ave “Jane Doe is the poorest person living on Moneybags Avenue.” “Joe Shmoe is the wealthiest person living on Poverty Blvd.” Slide18:  “San Francisco ranks 55 out of 58 counties when it comes to state and national performance measures…” SF Chronicle, “No refuge. For Foster youth, it’s a state of chance”, November 15, 2005 Slide19:  San Francisco: AB636 UCB State Measures (Used in NCYL Ranking) % IMPROVEMENT Jan ‘04 compared to June ‘06 (+) indicates a measure where a % increase equals improvement. (-) indicates a measure where a % decrease equals improvement. indicates a measure where performance declined. “San Francisco ranks 55 out of 58 counties when it comes to state and national performance measures…” SF Chronicle Rankings mask improvement over time. However, even improvement over time and relatively high rankings can be misleading. 2) Compare Apples and Oranges:  2) Compare Apples and Oranges Two doctors in Anytown, CA… Doctor #1 Doctor #2 What if the populations served by each doctor were very different? 2/1000 20/1000 Doctor of the Year? Slide21:  “Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.” SF Chronicle, “Accidents of Geography”, March 8, 2006 “Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”:  “Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.” Different families and children served? Different related outcomes? First entry rates in Fresno are consistently lower Re-entries in Fresno are also lower… 3. Other considerations… Resources available, resource allocation choices Performance trends over time 3) Data snapshots…:  Number of Crimes Period 1: 76 Period 2: 51 Period 3: 91 Period 4: 76 3) Data snapshots… Crime jumped by 49%!! No change. Crime dropped by 16% Average = 73.5 Crime in Anytown, CA… Slide24:  “A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area...” SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005 “A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…”:  Abuse in Care Rate Period 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00% Responsible use of the data prevents us from making any of these claims (positive or negative). The sample is too small; the time frame too limited. “A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…” 100% improvement! 0 Children Abused! = 2/111 = 0 = 2/122 = 1/119 4) Unrepresentative findings…:  4) Unrepresentative findings… Survey of people in Anytown, CA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium… But what if you were then told that respondents had been sampled from a list of season football ticket holders? Slide27:  “Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” “My Word”, Oakland Tribune, May 25, 2006 “…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune:  “…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune Factually true? Yes. Misleading? Yes. This was a survey of emancipated foster youth Emancipated youth represent a distinct subset of the foster care population This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care… 5) Logical “Flipping”…:  5) Logical “Flipping”… Headline in The Anytown Chronicle: 60% of violent crimes are committed by men who did not graduate from high school. “Flip” 60% of male high school drop-outs commit violent crimes? Slide30:  “One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.” Bath and Haapala, 1993 as cited in “Shattered bonds: The color of child welfare” by Dorothy Roberts “One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.” :  “One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.” In reading statistics such as the above, there is a tendency to want to directionally “Flip” the interpretation But the original and flipped statements have very different meanings! 75% of neglect cases involved families with incomes under $10,000 DOES NOT MEAN 75% of families with incomes under $10,000 have open neglect cases Put more simply, just because most neglected children are poor does not mean that most poor children are neglected 6) False Causality…:  6) False Causality… A study of Anytown residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Finding an association between two factors does not mean that one causes the other… Hair Length Height Gender X Slide33:  “A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…” As reported in Sidebotham and Heron’s 2006 article “A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…” :  “A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…” Should one conclude that prematurity is a causal factor in maltreatment? (Drug use?) a third factor prematurity maltreatment California’s Response to Data Misuse?:  California’s Response to Data Misuse? CA has had the will to weather the storm(s)… Continued efforts to frame the data, educate interested media, policymakers, and others What do these findings mean? How can these data be used to gain insight into where improvements are needed? Counties have been proactive in discussing both the “good” and the “bad” (be first, but be right). Be transparent If not playing offense…playing defense Data still public!! (Thank you to the CWDA for these bullets!) Slide36:  CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Needell, B., Webster, D., Armijo, M., Lee, S., Cuccaro-Alamin, S., Shaw, T., Dawson, W., Piccus, W., Magruder, J., Exel, M., Conley, A., Smith, J. , Dunn, A., Frerer, K., & Putnam-Hornstein, E., (2007). Child Welfare Services Reports for California. Retrieved [month day, year], from University of California at Berkeley Center for Social Services Research website. URL: <http://cssr.berkeley.edu/UCB_CHILDWELFARE/> Barbara Needell [email protected] Emily Putnam-Hornstein [email protected]

Related presentations


Other presentations created by Susett

India presentation
31. 01. 2008
0 views

India presentation

deltas with changes
09. 01. 2008
0 views

deltas with changes

kosovo
09. 01. 2008
0 views

kosovo

Phthalates
10. 01. 2008
0 views

Phthalates

mathsproblems
10. 01. 2008
0 views

mathsproblems

predicateAndArgument
10. 01. 2008
0 views

predicateAndArgument

barrie jpc06
16. 01. 2008
0 views

barrie jpc06

gama pgb 18mar03 nofigs
14. 01. 2008
0 views

gama pgb 18mar03 nofigs

Conducting a Hazard Assessment
20. 01. 2008
0 views

Conducting a Hazard Assessment

gravity07
22. 01. 2008
0 views

gravity07

Jtaylor99
22. 01. 2008
0 views

Jtaylor99

Witches part1
04. 02. 2008
0 views

Witches part1

SQG
04. 02. 2008
0 views

SQG

pedmalls
04. 02. 2008
0 views

pedmalls

16janellerose
17. 01. 2008
0 views

16janellerose

plant exercise
11. 02. 2008
0 views

plant exercise

Moira Chan Yeung ENG PTT
07. 02. 2008
0 views

Moira Chan Yeung ENG PTT

303lec27
15. 01. 2008
0 views

303lec27

philanthrony
18. 02. 2008
0 views

philanthrony

PHYS1311 101706 bw
11. 01. 2008
0 views

PHYS1311 101706 bw

MRCME Sciatica
28. 02. 2008
0 views

MRCME Sciatica

Mr I Noy
05. 03. 2008
0 views

Mr I Noy

twri 06
12. 01. 2008
0 views

twri 06

Garbage
11. 01. 2008
0 views

Garbage

2005110803450707
21. 01. 2008
0 views

2005110803450707

sres blended 2006
19. 03. 2008
0 views

sres blended 2006

State Contract Airfares
21. 03. 2008
0 views

State Contract Airfares

EU eurosceptic view
22. 01. 2008
0 views

EU eurosceptic view

lozen
12. 02. 2008
0 views

lozen

Manatee
03. 04. 2008
0 views

Manatee

InfoDay2 IST
28. 03. 2008
0 views

InfoDay2 IST

20071107 TECO 3Q07 v4
08. 04. 2008
0 views

20071107 TECO 3Q07 v4

vgtu prezentacija
16. 04. 2008
0 views

vgtu prezentacija

influensa051110
07. 02. 2008
0 views

influensa051110

tara 2008 content area reading
11. 03. 2008
0 views

tara 2008 content area reading

schultz testbed overview lecture
23. 04. 2008
0 views

schultz testbed overview lecture

305
24. 04. 2008
0 views

305

C5 trade2
07. 05. 2008
0 views

C5 trade2

Chapman2
02. 05. 2008
0 views

Chapman2

200611 Wolski DampingRings
25. 02. 2008
0 views

200611 Wolski DampingRings

com611
13. 01. 2008
0 views

com611

ProphetMuhammad PBUH
04. 02. 2008
0 views

ProphetMuhammad PBUH

Chong Roots
06. 02. 2008
0 views

Chong Roots

ene29 11 02
23. 01. 2008
0 views

ene29 11 02

partnersinaredesignn etwork
13. 02. 2008
0 views

partnersinaredesignn etwork

cotton2
24. 01. 2008
0 views

cotton2

New Apps 2006A
24. 01. 2008
0 views

New Apps 2006A

2005funding
07. 03. 2008
0 views

2005funding

nbi03 rangod
16. 01. 2008
0 views

nbi03 rangod

Roadshow 6 03 AM
22. 01. 2008
0 views

Roadshow 6 03 AM