sigcse97

Information about sigcse97

Published on December 10, 2007

Author: Edolf

Source: authorstream.com

Content

Online Evaluation in WWW-based Courseware: The QUIZIT Project:  Online Evaluation in WWW-based Courseware: The QUIZIT Project Lúcio Cunha Tinoco Edward A. Fox N. Dwight Barnette Virginia Tech Background and Motivation:  Background and Motivation EI project, Courseware, Interactive Learning with a Digital Library in CS - http://ei.cs.vt.edu Workshop June 15-21 (apply to [email protected]) Local and Distance Learning Evaluation including based on server’s use Large enrollment, grading overhead Personalized System of Instruction (PSI) QUIZIT: Goals, Applications:  QUIZIT: Goals, Applications Fully WWW-based, automated test system supplements other types of evaluation is not viewed as a substitute for other types for open-book or monitored quizzes helps in identifying problem areas focuses on task completion, not on scores For PSI and/or large enrollment courses Reduces administrative and grading overhead QUIZIT: Goals, Applications:  QUIZIT: Goals, Applications Integration w/ WWW-based courseware and digital libraries Reusable content Automatic grading w/ high interactivity Instant feedback Alternative questions Easy to use and to apply Test Systems:  Test Systems Traditional: letter, paper, oral Test formats Turn-around, grading time Electronic submission, manually graded Video conference, e-mail, etc. Reduction in turn-around, same grading time Electronic submission, automatically graded CBTs, WBTs, Hybrids Reduction in turn-around and grading time Existing Automated Test Systems: Denominations:  Existing Automated Test Systems: Denominations CBTs Authorware Toolbook Electronic Trainer WBTs Mklesson LXRTest Hybrids: PC authoring, option of Internet deployment (ex: Toolbook II) Existing Test Systems: Features:  Existing Test Systems: Features Authoring interface Usability Document representation, portability, reusability Execution environment Distributed vs. stand-alone Record-keeping Adaptability based on performance? static (e.g., predefined) vs dynamic (e.g., inference) Existing Test Systems: CBTs, Hybrids:  Existing Test Systems: CBTs, Hybrids Proprietary interfaces powerful, but hard to learn each system represents documents differently hard to integrate w/ digital libraries compromises portability Stand-alone environments Hybrids provide Internet deployment, but do not provide group capabilities usually do not have the concept of a “class” Existing Test Systems: WBTs:  Existing Test Systems: WBTs Interfaces only fair-to-good Limitations in adding adaptability Presentational syntax in document representation (poor logical modeling, ad hoc use of HTML-extensions) Poor record-keeping and monitor capabilities Also do not treat group monitoring, lack concept of “class” QUIZIT System Architecture:  QUIZIT System Architecture QUIZIT: Main Features:  QUIZIT: Main Features Use of standarized authoring language SGML-based QUIZIT Markup Language (QML) But still provides all the power of HTML and Java Tests can be adaptable Complete log of results and actions into a database Monitor interface in the WWW QUIZIT: Modules:  QUIZIT: Modules Authoring Monitor Registration Activity Run-time Registration and authentication Grading and feedback Review The QUIZITAuthoring Module: QML:  The QUIZITAuthoring Module: QML Content-based representation Use of SGML (ISO/IEC 8879) to define QML Avoids presentational elements The QUIZITAuthoring Module: QML (top):  The QUIZITAuthoring Module: QML (top) The QUIZITAuthoring Module: QML (middle):  The QUIZITAuthoring Module: QML (middle) The QUIZITAuthoring Module: QML (end):  The QUIZITAuthoring Module: QML (end) The QUIZIT Authoring Module: QML Translation:  The QUIZIT Authoring Module: QML Translation After instructor prepares the QML file The QUIZIT Monitor Module: Record-keeping:  The QUIZIT Monitor Module: Record-keeping Log student accesses to the system Keep track of individual and class performance on quizzes Can change system state and db at any time if user is authorized to do so (instructor, GTA). Also includes student registration. Instructor mSQL Database Engine:  mSQL Database Engine Stores and retrieves students’ records and accesses to the system IDs and passwords Scores and students’ answers on every quiz (for every unit in the course) Access logs (begin, end time) Last quiz taken Next quiz to be taken The QUIZIT Run-time Module: Registration and Authentication:  The QUIZIT Run-time Module: Registration and Authentication Provides online password registration and distribution (by Tommy Johnson) Passwords are encrypted in the server side (is client side encryption feasible?) Authenticates every user once and then uses session cookie throughout the quiz (use of hidden fields) The QUIZIT Run-time Module: The Grader Engine:  The QUIZIT Run-time Module: The Grader Engine After selecting a course, authenticating, selecting a unit and taking the quiz... Student using a WWW browser • end time • scores • answers • next quiz The QUIZIT Run-time Module: Reviewing Records and Old-Quizzes:  The QUIZIT Run-time Module: Reviewing Records and Old-Quizzes At any given point in time students can: Review old quizzes (student may not retake quizzes for credit) Come back to the system and re-start from where they left Review their records submission times scores (cumulative, per quiz) the next quiz they will have to take QUIZIT in action!:  QUIZIT in action! Graduate-level course: CS5604 (11 students) Personalized System of Instruction (PSI) Course divided into 11 units Each unit has one quiz with two levels. Each level has two remedial quizzes score < cutoff score < cutoff score > = cutoff CS5604: Progress Report:  CS5604: Progress Report CS5604: Progress: Conclusions :  CS5604: Progress: Conclusions Students do procrastinate They tend to procrastinate more in middle units Keep the pace in the beginning Catch up in the end because of deadline Big quiz “rush” in the end of the semester CS5604: Student Performance:  CS5604: Student Performance CS5604: Performance: Conclusions:  CS5604: Performance: Conclusions Most students only go through 2 of 6 levels “Harder” quizzes: IF, MM, KB Some questions? Does procrastination lead to worse performance? Does “deadline fever” lead to worse performance? QUIZIT in action! (cont.):  QUIZIT in action! (cont.) Undergraduate Level Course: CS1604 (about 150 students) Each quiz has just one level Security concerns are much greater Password distribution needs to be automated How can we assure that one student is not taking a quiz for another? CS1604: Progress and Performance:  CS1604: Progress and Performance Students procrastinate even more Less mature? Performance was based on scores rather than task completion Flat quiz structure may lead to poor indicators Evaluation and Conclusion: HCI Aspects:  Evaluation and Conclusion: HCI Aspects Authoring quizzes with QUIZIT takes a little more time than traditional quizzes Huge savings in grading time, though... Security and authentication are critical in large enrollment classes Extensive record keeping and WWW monitor interfaces are very important Change in the role of the GTA Evaluation and Conclusion: Technical Aspects:  Evaluation and Conclusion: Technical Aspects Instructors need training in QML CGI/mSQL is a very nice combination ... but new Java features may complement this model in the future Record keeping can be further extended Further security mechanisms need to be developed (client side encryption, camera monitoring, etc.) for more general applications Evaluation and Conclusion: Pedagogical Aspects:  Evaluation and Conclusion: Pedagogical Aspects QUIZIT was not designed to completely substitute for other types of evaluation Problems w/ multiple-choice exams can’t be negleted Be careful when using scores in adaptable quizzes! Focus should be on task completion Should work well w/ self-evaluation and distance learning Evaluation and Conclusion: Pedagogical Aspects:  Evaluation and Conclusion: Pedagogical Aspects Does QUIZIT/PSI promote procrastination? What can we do to avoid procrastination in PSI? One option: Bonus for tasks completed in a certain time limit How can we explore more in quiz adaptability? Future Developments:  Future Developments Improvements in student feedback Better error report after each test Graphical User Authoring Interface but keep QML representation Improve question modeling Add more pattern matching options for fill-in questions Training program, Packaging, Distribution

Related presentations


Other presentations created by Edolf

Chapter 02
30. 12. 2007
0 views

Chapter 02

Sauli GEM
29. 10. 2007
0 views

Sauli GEM

IV Classical
26. 11. 2007
0 views

IV Classical

flex builder debugger
28. 11. 2007
0 views

flex builder debugger

Powell
04. 12. 2007
0 views

Powell

CLC History Part I
29. 10. 2007
0 views

CLC History Part I

ActivityDiagrams
06. 11. 2007
0 views

ActivityDiagrams

Perriono Helsinki TFMM
01. 11. 2007
0 views

Perriono Helsinki TFMM

IT
06. 11. 2007
0 views

IT

GKII005
16. 11. 2007
0 views

GKII005

Using Doubles in Math
16. 11. 2007
0 views

Using Doubles in Math

Creative Dance
23. 11. 2007
0 views

Creative Dance

Opac Tutorial Module 3
26. 11. 2007
0 views

Opac Tutorial Module 3

LaszloAndWebOrb
28. 11. 2007
0 views

LaszloAndWebOrb

McCarthy Hearings
24. 12. 2007
0 views

McCarthy Hearings

International politics
28. 12. 2007
0 views

International politics

InnoArdentBridgesABC 04 01 23 07
01. 01. 2008
0 views

InnoArdentBridgesABC 04 01 23 07

class presentation
02. 01. 2008
0 views

class presentation

420 07 lect2
03. 01. 2008
0 views

420 07 lect2

stiles
03. 01. 2008
0 views

stiles

Civic Engagement
07. 01. 2008
0 views

Civic Engagement

How to Make Biodiesel D Geller
12. 11. 2007
0 views

How to Make Biodiesel D Geller

The Agricultural World Part I
27. 11. 2007
0 views

The Agricultural World Part I

VUS06 07
12. 12. 2007
0 views

VUS06 07

6 Choice of Law
31. 10. 2007
0 views

6 Choice of Law

page9 1
20. 02. 2008
0 views

page9 1

ACCRA PP Show June
24. 02. 2008
0 views

ACCRA PP Show June

631pub04 sec12videoMPEG
27. 02. 2008
0 views

631pub04 sec12videoMPEG

who1105
27. 03. 2008
0 views

who1105

NCSA
06. 11. 2007
0 views

NCSA

Introduzione
21. 11. 2007
0 views

Introduzione

Lect 23
14. 12. 2007
0 views

Lect 23

EAPN 0604 turkey2
28. 11. 2007
0 views

EAPN 0604 turkey2

8 NASS Aug Rpt
04. 10. 2007
0 views

8 NASS Aug Rpt

caca ao tesouro
31. 12. 2007
0 views

caca ao tesouro

Lec3 Ling
19. 11. 2007
0 views

Lec3 Ling

Corrie
22. 11. 2007
0 views

Corrie

Ancient Greece and Rome
30. 10. 2007
0 views

Ancient Greece and Rome