Workshop_Designing and Evaluating the Impact of Your Program

Information about Workshop_Designing and Evaluating the Impact of Your Program

Published on July 14, 2014

Author: csimkovich

Source: authorstream.com

Content

PowerPoint Presentation: Designing and Evaluating the Impact of your Program: A Case Study Sue Wagner , M.Ed. Vice President Education and Information The Morton Arboretum, Lisle, IL Amy Padolf , M.Ed. Director of Education Fairchild Tropical Botanic Garden, Coral Gables, FL Lorrie Beaumont, Ed.D . Evergreene Research and Evaluation Sugar Grove, IL PowerPoint Presentation: Sue Wagner, M. Ed. Vice President Education and Information The Morton Arboretum, Lisle, IL PowerPoint Presentation: Amy Padolf , M.Ed. Director of Education Fairchild Tropical Botanic Garden Coral Gables, FL PowerPoint Presentation: Lorrie Beaumont, Ed.D . Director - Evergreene Research and Evaluation Sugar Grove, IL The M&Ms of our workshop: Choose one M&M from the bowl on the table. Respond to the question that matches the color of your M&M Red: What does it M ean to conduct evaluation? Green: Why does evaluation M atter? Yellow: What are some evaluation M ethods you are familiar with? 5 The M&Ms of our workshop Introduction: Key understanding: Evaluation is an integral part of program design: It informs program development and provides evidence of program impact. Outcomes for today: Gain insight on planning and evaluating a program/exhibit using a real case. Participate in the creation of a model for program development and evaluation. 6 Introduction What is evaluation?: “ To evaluate something means to determine its merit , worth , value or significance . ” (Michael Quinn Patton) It is a process Purpose is to yield information about how well an intervention is working Should be an integral part of an activity or program from the beginning 7 What is evaluation? Why conduct evaluation?: WHAT? SO WHAT? NOW WHAT? 8 Why conduct evaluation? Evaluation process: Step 1: Determine what needs to be evaluated and why Step 2: Develop a logic model (conceptual model) Step 3: Develop evaluation questions Step 4: Determine best data collection methods Step 5: Analyze the data Step 6: Report the results with recommendations 9 Evaluation process A little background about evaluation: Front End/Needs Assessment Formative/Prototyping Remedial – the bridge Summative – Did you do what you set out to do? Developmental evaluation 10 A little background about evaluation Step 1: Determine what needs to be evaluated and why : The Fairchild Challenge Evaluation question: The Fairchild Challenge examples (2007-08 study): To what extent is The Fairchild Challenge program achieving its stated priorities? How do key program stakeholders/teachers perceive the program? What unanticipated outcomes has the program had? 11 Step 1: Determine what needs to be evaluated and why The Fairchild Challenge ENVIRONMENTAL IMMERSION DAYS : The Fairchild Challenge ENVIRONMENTAL IMMERSION DAYS Step 2 -Logic Models: Thinking “evaluatively”: Describes the “ theory of change ” underlying an intervention (program) A tool and an approach for depicting the critical elements in a project and identifying where evaluation is most important. Takes practice 19 Step 2 -Logic Models: Thinking “ evaluatively ” Outcomes (Impacts): Awareness, Knowledge, Understanding (of) Engagement or Interest (in) Attitude (towards) Behavior (related to) Skills (based on) Other (project specific) www.informalscience.org /evaluations/ eval_framework.pdf 20 Outcomes (Impacts) Step 3: Environmental Immersion Days Evaluation Questions: To what extent and in what ways are we impacting students ’ post-secondary education and career choices? To what extent and in what ways are we affecting change in knowledge, behavior or interests? To what level?   Step 3: Environmental Immersion Days Evaluation Questions Example of Fairchild Challenge Immersion Days Program Model: Purpose: To offer high school students an interactive experience in which they spend the day with Fairchild scientists, university researchers and local conservation biologists to explore major fields of study and careers in biological and environmental science. Example of Fairchild Challenge Immersion Days Program Model PowerPoint Presentation: Program service : A field day at Fairchild where students self select a particular biological and environmental science field of study to investigate. Students will rotate through several stations that focus on topics within that field of study where they participate in authentic scientific research with professionals; An opportunity for university faculty and community researchers to recruit potential dynamic, self-motivated students for their labs. PowerPoint Presentation: Outcomes (Impact: knowledge, interest, awareness) Students will identify an interesting career they might want to explore. Students will have an increased interest in pursuing a career in the biological, environmental or ecological sciences. Students become aware of research being conducted in a particular field of study that interests them. Students will make useable connections with professionals in the field. Students will develop a clear pathway towards continued education and careers in biological and environmental sciences. Faculty will make connections with students interested in their field of study for which they can recruit for their university. PowerPoint Presentation: Indicators 300 local high school students participate annually in two separate workshop days 85% of participants submit a written report detailing their experience 80% of participating students indicate an interest in pursuing a career in biological, environmental, or ecological science 65% of participating students identify a particular field of science they would like to explore Each participating faculty member identifies at least 2 students to continue to mentor. Step 4: Determine best data collection methods?: Qualitative vs. Quantitative Sampling/Choosing respondents: From whom will you collect the data? Adults Children Volunteers Non-visitors When will you collect the data? Where will you collect the data? What resources will you need to collect data? (clip boards, thank you letters, incentives, tables and chairs, volunteers…) 26 Step 4: Determine best data collection methods? Some methods you might choose: Observation Interview Focus Groups Survey/Questionnaire Reflective journals 27 Some methods you might choose The Fairchild Challenge Immersion Days Method Ideas: Student surveys Questionnaires Student and teacher feedback sessions Students sharing session during lunch Student written reports 28 The Fairchild Challenge Immersion Days Method Ideas Step 5: Analyze the data: Analysis depends on whether it was a mainly qualitative or quantitative study How shall we analyze The Fairchild Challenge data Based on what you ’ ve learned today, what methods would you consider as you analyze data? 29 Step 5: Analyze the data Step 6: Report and use the results: Best way to report The Fairchild Challenge data Recommendations? 30 Step 6: Report and use the results Evaluation process: Step 1: Determine what needs to be evaluated and why Step 2: Develop a conceptual model (logic model) Step 3: Develop evaluation questions Step 4: Determine best data collection methods Step 5: Analyze the data Step 6: Report the results with recommendations. 31 Evaluation process How can you start evaluating tomorrow?: What did you see? What did you hear? Who should you tell? How do you move from anecdotal to real data? How will your observations improve your program? 32 How can you start evaluating tomorrow? Example: Exhibit/Program portfolios: Logic model Photographs Video Drawings Field notes Observations Transcripts of interviews Meeting notes 33 Example: Exhibit/Program portfolios Evaluation tools : www.visitorstudies.org www.informalscience.org www.informalcommons.org 34 Evaluation tools Key Takeaways: Key understanding: Evaluation is an integral part of program design: It informs program development and provides evidence of program impact. Gained insight on planning and evaluating a program/exhibit Practiced using a logic model for program development and evaluation Participated in planning an evaluation study for the Fairchild Challenge Recognize the importance of evaluation as part of a program. 35 Key Takeaways Questions?: Exit ticket Topics for exploration 36 Questions? Resources: Lorrie Beaumont [email protected] Sue Wagner [email protected] Amy Padolf [email protected] 37 Resources

Related presentations


Other presentations created by csimkovich

AES Lunch_Presentation 2 of 2
15. 07. 2014
0 views

AES Lunch_Presentation 2 of 2

AES Lunch_Presentation 1 of 2
15. 07. 2014
0 views

AES Lunch_Presentation 1 of 2

APGA Overview
23. 09. 2014
0 views

APGA Overview

Session 2 Fundraising Sucess
23. 09. 2014
0 views

Session 2 Fundraising Sucess

Gould_APGA
22. 10. 2014
0 views

Gould_APGA

Traucht_APGA
22. 10. 2014
0 views

Traucht_APGA

HeritageWinterthur
22. 10. 2014
0 views

HeritageWinterthur

APGA2014Winterthur
22. 10. 2014
0 views

APGA2014Winterthur

SBBG APGA Historic Landscapes
22. 10. 2014
0 views

SBBG APGA Historic Landscapes

Conrad Session 5 Collections-CBD
13. 11. 2014
0 views

Conrad Session 5 Collections-CBD

Kinard Session 5 Collections-CBD
13. 11. 2014
0 views

Kinard Session 5 Collections-CBD

Intro Session 5 Collections-CBD
13. 11. 2014
0 views

Intro Session 5 Collections-CBD

Husby Session 5 Collections-CBD
13. 11. 2014
0 views

Husby Session 5 Collections-CBD

APGA 2015 Tourism Panel FINAL
10. 08. 2015
0 views

APGA 2015 Tourism Panel FINAL