2007 Annual Session Michael Quinn Patton

Information about 2007 Annual Session Michael Quinn Patton

Published on January 11, 2008

Author: Carmela

Source: authorstream.com

Content

Deepening:  Deepening Extension’s Knowledge Base Seeing Through an Evaluation Lens:  Seeing Through an Evaluation Lens Slide3:  In the beginning… Extension :  Extension as a knowledge-based learning organization Slide6:  LEARNING FRAMEWORK Action Beliefs Knowledge Premises:  Premises Action flows from some combination of beliefs/values and knowledge. Informed action has a strong knowledge base. By testing and evaluating beliefs, we build knowledge, thereby making our actions more informed and empirically based. Slide8:  In short, ACTIONS flow from some combination of… BELIEFS and KNOWLEDGE . Learning Organization:  Learning Organization Moving from beliefs to knowledge as the foundation for and basis of action. Slide10:  What constitutes high quality, useful knowledge? Aspirations:  Aspirations Scientifically-based knowledge Research-validated findings Evidence-based practice Best practices/Promising practices Generalizable Lessons Learned Slide12:  The Debate About Randomized Controls in Evaluation: The Gold Standard Question GOLD STANDARD: METHODOLOGICAL APPROPRIATENESS not Methodological orthodoxy or rigidity:  GOLD STANDARD: METHODOLOGICAL APPROPRIATENESS not Methodological orthodoxy or rigidity Slide14:  Observational Data Suffices A study in the British Medical Journal by Gordon Smith and Jill Pell (2003) found that…. http://bmj.bmjjournals.com/cgi/content/full/327/7429/1459?ck=nck Slide15:  There have been NO randomized control trials of parachute use. Slide17:  Smith and Pell concluded: “Only two options exist. The first is that we accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of an intervention.” OR Slide18:  those who criticize interventions that lack a RCT evidence base will not hesitate to demonstrate their commitment by volunteering for a double blind, randomized, placebo controlled, crossover trial.” What is possible and appropriate::  What is possible and appropriate: Multiple sources of data about each proposition Triangulation of sources High Quality Lessons Slide20:  The Metaphor of Triangulation Multiple Data Sources:  Multiple Data Sources to build Knowledge Highly Rated Knowledge:  Highly Rated Knowledge “Star” Sources:  “Star” Sources basic and applied research Limitations to published research as the sole or primary source of knowledge:  Limitations to published research as the sole or primary source of knowledge Errors Inadequate peer review processes Corrections over time Time for replication Generalizability issues Corruption: publishing pressure, funding pressures “Star” Sources:  “Star” Sources basic and applied research evaluation findings --patterns across programs “Star” Sources:  “Star” Sources basic and applied research evaluation findings--patterns across programs practice wisdom and experience of practitioners: individuals & teams “Star” Sources:  “Star” Sources basic and applied research evaluation findings--patterns across programs practice wisdom and experience of practitioners experiences and knowledge of consumers, program participants, clients, intended beneficiaries… More “Star” Sources:  More “Star” Sources expert opinion “Star” Sources:  “Star” Sources expert opinion cross-disciplinary connections and patterns “Star” Sources:  “Star” Sources expert opinion cross-disciplinary connections and patterns THEORY – capturing and organizing knowledge The Theory-Practice Connection:  The Theory-Practice Connection There’s nothing so practical as a good theory. Lazarsfeld Slide32:  Theory of Change Program Theory Logic Models Theory of Action Intervention Strategy Theory-driven evaluation Focus of High Quality Lessons Learned & Knowledge::  Focus of High Quality Lessons Learned & Knowledge: Enhancing Our Theory of Change High Quality Lessons :  High Quality Lessons Those lessons that are supported by and triangulated from multiple sources that reinforce and affirm the knowledge encompassed in the lesson, yielding: Slide35:  High Quality, Triangulated Knowledge to inform a Theory of Change Assessing Lessons:  Assessing Lessons Assess the importance of the lesson learned for a particular action Examine and evaluate the strength of the connection to outcomes attainment. Compare case examples to extract principles. Reflective Practice Example:  Example Extension’s Mission and Theory of Change Slide38:  LEARNING FRAMEWORK Action Beliefs Knowledge Slide39:  Extension’s Mission BELIEF DEBATE: Knowledge Transfer (Education) or Slide40:  Knowledge Transfer (Education) or Behavior & Practice Change Bennett’s Hierarchy:  Bennett’s Hierarchy 7. End results --impact on the overall problem   6. Practice and behavior change   5. Knowledge, attitude, and skill changes 4. Reactions & satisfaction  3. Participation  2. Activities 1. Inputs & resources Basic Education Model Belief:  Basic Education Model Belief Changed Behavior Knowledge change Basic Assumptions:  Basic Assumptions Rational, information-based decision-making and Linear, direct cause-effect outcomes model Reality-testing:  Reality-testing Poor information processors, beliefs trump evidence Selective perception Humans as emotional animals Neuroeconomics Evidence:  Evidence Evaluation findings – knowledge alone generally doesn’t work to produce lasting behavior and practice changes for difficult and major issues Evidence:  Evidence Research findings – lasting behavior change is multi-faceted, requiring significant incentives, on-going reinforcement of new behaviors, and integration with other beliefs, values, and knowledge Evidence:  Evidence Theory – Systems theory emphasizes understanding behavior change in a systems context, that is, in relation to other factors and dynamics, not as an isolated phenomenon: Farming systems Family systems work Complex adaptive systems approaches Other Evidence:  Other Evidence Feedback from clients and program participants Expert opinion Cross-disciplinary findings from -- behavioral economics -- systems psychology -- organizational sociology -- adult learning Slide49:  The nature of EXPERTISE The Knowledge Challenge::  The Knowledge Challenge: Matching the knowledge transfer and program intervention strategy to the nature of the situation, the desired changes, system understandings, resources, and timeline to optimize impact within a local knowledge framework. Some premises::  Some premises: Evaluation is part of initial program design, including conceptualizing the theory of change Evaluator’s role is to help users clarify their purpose, hoped-for results, and change model. Evaluators can/should offer conceptual and methodological options. Evaluators can help by questioning assumptions. Conceptual Options:  Conceptual Options Simple Complicated Complex Types of Community Issues The Stacey Matrix:  Types of Community Issues The Stacey Matrix Certainty Agreement Close to Far from Far from Close to Simple:  Simple Certainty Agreement Close to Far from Far from Close to Simple Plan, control Slide55:  Following a Recipe A Rocket to the Moon Raising a Child Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple Technically Complicated:  Technically Complicated Certainty Agreement Close to Far from Far from Close to Simple Plan, control Technically Complicated Experiment, coordinate expertise Slide57:  Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple Socially Complicated:  Socially Complicated Certainty Agreement Close to Far from Far from Close to Simple Plan, control Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Socially complicated:  Socially complicated Implementing human rights agreements, like gender equity or outlawing child labor Environmental Initiatives Many different and competing stakeholders Diverse vested interests High stakes Socially complicated:  Socially complicated situations pose the challenge of coordinating and integrating many parts Know When Your Challenges Are In the Zone of Complexity:  Know When Your Challenges Are In the Zone of Complexity Certainty Agreement Close to Far from Far from Close to Simple Plan, control Zone of Complexity Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Systems Thinking Relationship Building Collaboration Good Enough Vision Chunking Around Drivers Minimum Specifications Multiple Actions Adaptability & Organic Slide62:  Following a Recipe A Rocket to the Moon Raising a Child Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Every child is unique Uncertainty of outcome remains Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipes produce standard products Certainty of same results every time Simple Know When Your Challenges Are In the Zone of Complexity:  Know When Your Challenges Are In the Zone of Complexity Certainty Agreement Close to Far from Far from Close to Simple Plan, control Zone of Complexity Technically Complicated Experiment, coordinate expertise Socially Complicated Build relationships, create common ground Chaos Massive Avoidance Slide64:  Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical and necessary Sending one rocket increases assurance that next will be ok High level of expertise in many specialized fields + coordination Separate into parts and then coordinate Rockets similar in critical ways High degree of certainty of outcome Formulae have only a limited application Raising one child gives no assurance of success with the next Expertise can help but is not sufficient; relationships are key Can’t separate parts from the whole Every child is unique Uncertainty of outcome remains Complicated Complex The recipe is essential Recipes are tested to assure replicability of later efforts No particular expertise; knowing how to cook increases success Recipe notes the quantity and nature of “parts” needed Recipes produce standard products Certainty of same results every time Simple Slide65:  Challenges: Situation Recognition and Knowledge Appropriate to the Situation Slide66:  Systems Parts are interdependent such a change in one part changes all parts The whole is greater than the sum of the parts Focus on interconnected relationships Systems are made up of sub-systems and function within larger systems Slide68:  Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens 2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome) Slide69:  Systems web showing possible influence linkages to a pregnant teenager Slide70:  Program systems web showing possible institutional influences affecting pregnant teenagers' attitudes and behavior Slide71:  Sustainable change: Systems dynamic reinforcing feedback loops Slide72:  Using Different System Lenses to understand a “particular” System Biologic System Energy Well-being Cycle status Vitality Sociologic System Relationships Conversations Interdependence Status issues Meaning/sense Environment Volatility Predictability Spatial Proximities Diversity Economic System Resources Fears & rewards Aspiration Political System Power Connections Possibility Locus of control Anthropologic System Values Culture/Milieu Information System Education Speed & access Fidelity/utility Knowledge Storage Psychological System Incentives Heuristics Reinforcements Behavior Change Slide73:  Extension as a knowledge-based organization Sharing Knowledge:  Sharing Knowledge to change behavior and practice, and solve problems, based on a Theory of Change Slide75:  HYMN to KNOWLEDGE USE Sung to the tune of Auld Lang Syne Slide76:  May all Ex-ten-sion programs use Knowledge that consumers trust, That tell them how to separate What works from what’s a bust. We gather data near and far To see what we can learn The findings help us to decide What to share and what to burn. Slide77:  There comes a time for each of us When doubts may give us pause We wonder what results will last Will our knowledge turn out flawed? But be assured there's naught to fear Shared learning is what we seek Tri-angulate your sources now With authority you can speak. Slide78:  Our knowledge base we can affirm, Extension shares the truth. Our lessons learned we will apply On farms, in homes, with youth. Extension staff go forth as one, Researchers here stand tall, The world will be a better place Sharing knowledge is our call. References:  References Utilization-Focused Evaluation, 3rd. Edition, by Michael Q. Patton, Sage Publications, 1997. (New 4th edition forthcoming at the end of 2007.) Getting to Maybe: How the World Is Changed by Frances Westley, Brenda Zimmerman, and Michael Q. Patton, Random House Canada, 2006. http://www.randomhouse.ca/catalog/display.pperl?isbn=9780679314431 Slide80:  Systems Concepts in Evaluation – An Expert Anthology. 2006. Bob Williams and Iraj Imam AEA Monograph, EdgePress/AEA Point Reyes CA.

Related presentations


Other presentations created by Carmela

1 Introduction to Strategy
11. 01. 2008
0 views

1 Introduction to Strategy

2 Serge Frechette
07. 05. 2008
0 views

2 Serge Frechette

Erguden
02. 05. 2008
0 views

Erguden

Canada
24. 04. 2008
0 views

Canada

2007 freshman class
23. 04. 2008
0 views

2007 freshman class

sport
16. 04. 2008
0 views

sport

tamuraADI2004
08. 04. 2008
0 views

tamuraADI2004

Imagine Cup
03. 04. 2008
0 views

Imagine Cup

henry
28. 03. 2008
0 views

henry

IAU Laos
21. 03. 2008
0 views

IAU Laos

ijcnlp 20080109
19. 03. 2008
0 views

ijcnlp 20080109

Motivation
17. 01. 2008
0 views

Motivation

Chapter 05
09. 01. 2008
0 views

Chapter 05

stone masonry
10. 01. 2008
0 views

stone masonry

ppp David Asteraki Presentation
12. 01. 2008
0 views

ppp David Asteraki Presentation

Promer Materials 34
13. 01. 2008
0 views

Promer Materials 34

syntax 1 checklist
13. 01. 2008
0 views

syntax 1 checklist

ARChapter4
15. 01. 2008
0 views

ARChapter4

Gravity Control
16. 01. 2008
0 views

Gravity Control

PersuasionThroughRhe toric
17. 01. 2008
0 views

PersuasionThroughRhe toric

Math
21. 01. 2008
0 views

Math

050525Barlow
22. 01. 2008
0 views

050525Barlow

Chemistry PostGrad
22. 01. 2008
0 views

Chemistry PostGrad

ci avian influenza
23. 01. 2008
0 views

ci avian influenza

meteorology03
25. 01. 2008
0 views

meteorology03

04 iran
04. 02. 2008
0 views

04 iran

mobiquitous keynote
04. 02. 2008
0 views

mobiquitous keynote

Jay Patterson 9 15 05
11. 02. 2008
0 views

Jay Patterson 9 15 05

LCROSS OverviewforObs
08. 01. 2008
0 views

LCROSS OverviewforObs

Movies
29. 01. 2008
0 views

Movies

2007 Popcorn Presentation
07. 02. 2008
0 views

2007 Popcorn Presentation

ylp
18. 02. 2008
0 views

ylp

poster
25. 01. 2008
0 views

poster

squirrel
25. 02. 2008
0 views

squirrel

04 Greg McDougall
05. 02. 2008
0 views

04 Greg McDougall

F07C107 19Defense1
10. 01. 2008
0 views

F07C107 19Defense1

BeansB
16. 01. 2008
0 views

BeansB

Laser2003
18. 01. 2008
0 views

Laser2003

MHL 2005 06
13. 02. 2008
0 views

MHL 2005 06

dasso asse 2004
22. 01. 2008
0 views

dasso asse 2004

Cation biogeo chemistry
22. 01. 2008
0 views

Cation biogeo chemistry

04 1 IR Basics 3
14. 01. 2008
0 views

04 1 IR Basics 3

GSIYCF IYCF
15. 01. 2008
0 views

GSIYCF IYCF

Inspire HGL Version
29. 01. 2008
0 views

Inspire HGL Version

industriousnh
14. 03. 2008
0 views

industriousnh

su induction
07. 02. 2008
0 views

su induction