Seminar8Testing

Information about Seminar8Testing

Published on July 11, 2014

Author: sueswift

Source: authorstream.com

Content

Delta Module One Consolidation Seminar Unit Eight: Delta Module One Consolidation Seminar Unit Eight Testing and Exams Terminology: Terminology Test qualities : validity (construct, content, face, predictive etc); reliability; practicality; backwash – positive/negative Test types : placement, diagnostic, progress, achievement, proficiency etc Item types : objective/subjective; discrete point /integrative; cloze, M/C, T/F etc etc Construct validity: Construct validity Why does the following test lack construct validity? Listening Comprehension Test A passage of 100 words will be dictated and you should write down exactly what you hear. The passage will be read once straight through, then each phrase will be repeated twice, slowly, with a pause to give you time to write. You will get one mark for each word which is correct (including spelling) Content validity: Content validity Which of the following dictations, included in an achievement test, has content validity for a group of learners who have been studying the past simple? Rachel Walters had a holiday with a difference last summer. She spent two months in Africa, where she worked as a volunteer… 2. Founded in 1863, Kemba is a small town situated on the east coast…. Face validity: Face validity Imagine that Cambridge ESOL decides to replace the listening comprehension questions in all its proficiency tests, (KET,PET, FCE, BEC etc) with a dictation. Would this have face validity? Predictive validity: Predictive validity The IELTS exam is intended to have predictive validity. Why? Reliability: Reliability Before 2008, the Delta written exam consisted of one 3hr paper, including three 1hr essay style questions. The new format is two 90min papers, with a compulsory break of 30-45 mins between them, with a total of nine questions, many of which are divided into unrelated sections (eg Paper 1 tasks 1, 2, 4). The new format should be more reliable. Why? Backwash: Backwash Which of the following are examples of backwash? A teacher is preparing a class for PET. The listening exam asks candidates to listen to dialogues and identify which picture, of several, relates to the dialogue. The teacher has chosen a textbook which contains a large number of listening activities with this format. A local university has made an IELTS score of 5.5 obligatory for admission to its degree courses. Local language schools has realised there is a market for IELTS courses and have started offering them. A class is preparing for an exam which is partially based on M/C grammar and lexis questions. Half way through the course the learners complained to the teacher that s/he never used this type of activity, and insisted that from now on one should be included in every lesson. Paper Two Task One: Paper Two Task One What do you have to do in this question? How long do you have for it? How many marks does it carry? Paper Two Task One: Paper Two Task One Time allotted :  20 mins Total marks available :  20 - One mark is identified for each positive / negative identified. An additional mark is given for each positive negative identified if its application to the learner is stated. -Up to two additional marks are awarded for consistently accurate use of testing terminology throughout the answer. The 14 marks are then weighted up to 20. The sample question: The sample question Type one of your positive points into the chatbox The sample question – points you could have made: The sample question – points you could have made  Level : The test is at a suitable level (BEC preliminary is intended for B1 learners). M will therefore find the test “doable” from this point of view, which will increase the reliability of the result. Points you could have made…: Points you could have made…  Instructions : There are clear instructions. M should know exactly what she has to do. This increases reliability . Points you could have made…: Points you could have made…  Task type : The test is a direct, integrative test of the writing skill, which is what the course should be focusing on. The tasks are business oriented and realistic, and one of them focuses on emails which is the text type which M has to produce. The test therefore has a certain amount of face validity (M is liable to believe it reflects her ability to write in English for business purposes) and could to a certain extent have positive backwash (keeping the course focused on business correspondance). Points you could have made…: Points you could have made…  Language : The test involves the use of discourse features, lexis and functions which M will/may have to use in her work related writing, namely - conventional features of emails such as initial and final salutations, and standard opening and closing phrases ( Thank you for your email/Please do not hesitate to contact me again if you need any further information ) - neutral to semi-formal style - expression of dates in writing - explaining how to reach the premises by public transport As the specific items mentioned may well have been covered on the course when dealing with her needs, the test has (at least partially) content validity . M is liable to believe that the test reflects the course she has done and is a fair test of her progress and of her ability to write in English. And the negative points?: And the negative points? Type one into the chatbox Points you could have made…: Points you could have made…  Communication vs Imagination : However, the context of the tasks is irrelevant to M’s work and this risks destroying its face validity for her. She may feel that asking her to perform a task relevant to a field (HR) that she knows nothing about and a situation she never faces (delegating tasks to a subordinate) in a format she has not studied (a memo) is unfair and will not give a result that reflects her true ability, ie is not reliable . This is particularly true of the first task where she has to “invent” tasks to write about in a memo. Points you could have made…: Points you could have made…  Fresh starts : There are only two sections to the test and therefore there is little opportunity for “fresh starts”.If real or perceived lack of knowledge hampers M’s performance on one of the sections, she will risk losing up to 50% of the marks. This decreases reliability . Points you could have made…: Points you could have made…  Test Type : BEC is a proficiency test which aims (over a much greater range of test items than used here) to ascertain a learner’s overall level. What is needed here is an achievement test which gives representative coverage to all and only the areas focused on during the course. This test does not do that. The test may therefore lack face validity for M, who will not understand its relevance to her course. Points you could have made…: Points you could have made…  Language : If the course has really focused on her needs, the test will not have content validity for two reasons : a) it does not cover many items which were in the course ; and b) many items which may not have been in the course are necessary for the test – eg memo format or using imperatives to delegate tasks : Don’t forget to /Make sure you... It will therefore not give an accurate picture of what she has achieved on the course. Points you could have made…: Points you could have made…  Evidence for the T/company : This lack of content validity means that the test will not give the necessary information concerning her ability to perform her work (ie the test will not have predictive validity ). Lack of this information in the report may result in potentially negative consequences for M such as being passed over for promotion or further training. Points you could have made…: Points you could have made…  Backwash : If on the other hand the teacher has “taught for the test” during the course, M. will be better prepared for the tasks, but will have wasted time focusing on areas of little importance to her, such as eg using imperatives to delegate tasks ( Don’t forget to/make sure that..). This negative backwash from the test will have made the course less relevant to her needs, meaning that she learns less of practical value to her and possibly makes her less contented with the course quality. And with different wording…: And with different wording… If the question had stated… A few days before the end of the course the company contact the school and say that they require a report. The teacher therefore… Which of the points covered previously would no longer be valid? What other point could you add? No longer valid…: No longer valid… The point concerning backwash. If the teacher had not yet decided to use the test, it could not have affected the course. Could be added…: Could be added… Practicality • The test has to be produced at short notice with no time for moderation or pre-testing . This test is easily available on the internet and, as it is produced by a reputable examining board, is presumably valid and reliable. It should therefore give a clear picture of M’s writing ability Could be added…: Could be added… Practicality The test has to be inserted into a course that has already been planned. Making time for it will therefore mean that something which M needs to do has to be omitted. This test would take no more than around 45 mins, thus causing minimum disruption to the course and ensuring that M covered as much of the planned content as possible. PowerPoint Presentation: This seminar is part of an on-line course run by Business Talk Milan for Cambridge ESOL Delta Module One. If you would like to find out more about our on-line and face-to-face EFL training courses, please contact us : [email protected]

Related presentations


Other presentations created by sueswift

Introductory seminar
06. 04. 2014
0 views

Introductory seminar

Unit 7 Nobel
26. 05. 2014
0 views

Unit 7 Nobel

OnlineConsolidationSeminarUnit1
28. 08. 2014
0 views

OnlineConsolidationSeminarUnit1

On-lineConsolidationSeminarUnit2
18. 10. 2014
0 views

On-lineConsolidationSeminarUnit2

Paper 2 Tasks 2 and 3
13. 11. 2014
0 views

Paper 2 Tasks 2 and 3

On-lineConsolidationSeminarUnit4
26. 04. 2015
0 views

On-lineConsolidationSeminarUnit4

The PDA Unrecorded
17. 09. 2015
0 views

The PDA Unrecorded

Intro to the Course Unrecorded
30. 06. 2016
0 views

Intro to the Course Unrecorded