Crafting a high quality examination paper

In this entry, I provide a quick, prescriptive guide to the production of examination papers. This is what I believe needs to be done to ensure that an exam paper is a good one:

  1. Make explicit the purposes of examinations in your module, unit, course or programme
  2. Compensate for the disadvantages of traditional examinations and capitalise upon their advantages
  3. Account for important principles in assessment quality
  4. Produce accurate exam papers with correct format, clear instructions and appropriate questions

Let’s look at these in turn…

1.Ask yourself the following questions:

Are exams an assessment method commonly used in your modules/programmes?

What are the rationales for employing exams rather than other assessment methods?

Are exams used in combination with other assessment methods?

2. Reflect on the pros and cons of exams:

  • Time efficient, cost effective
  • Diminish plagiarism and cheating
  • Staff/Student familiarity with exams
  • Motivate students to learn
  • Provide equal opportunity?
  • Relatively easy to mark
  • Provide data for performance analysis

How can you maximise the above advantages?

  • It is challenging to write good questions
  • Exams favour students skilled at doing exams
  • Promote surface learning and memorisation
  • Do not provide equal opportunity?
  • Are boring to mark
  • Do not result in much individual feedback for students

How can you compensate for the above disadvantages?

3. Principled assessment is…

  • for learning
  • inclusive
  • authentic
  • motivating
  • engaging
  • proportionate and manageable
  • transparent

Assessment methods are…

  • valid
  • reliable
  • consistent
  • fair
  • aligned with learning outcomes and teaching methods

Keep these principles in mind, do your best to balance them.

4. If your institution uses a standard format, check and follow it precisely.

To write clear instructions, use imperative mood as often as possible. For instance, instead of writing “You are reminded to write legibly and in complete sentences and avoid direct copying from the text.” write this instead, “Write clearly. Write in full sentences. Use your own words. Do not copy from the text.” It may sound overly simplistic, but the main aim is to explain the task. Keep the language simple.

For exam questions that include subject content, you can use a free online tool to check the vocabulary difficulty level that you are using: Compleat Web VocabProfiler https://www.lextutor.ca/vp/comp/

First, select NGSL + NAWL

NGSL

Copy and and paste your question text into the box and click SUBMIT. Then you will get a colour-coded report.

WebV

Basically, the blue words are the easiest, then the green, then the pink. The yellow words are general academic, i.e. they are common in many university subjects. The red words could be proper nouns or technical terms.

Using this online tool could help you reflect on how easy or difficult it is for your students to understand the question.

Be cautious when selecting instructional verbs because they can have more than one meaning. For example, compare the meanings of ‘explain’ in these two examples:

  • Explain the principles they follow in their practice of Corporate Governance.
  • Explain whether you think these practices can help the company to prevent having conflicts with other stakeholders.

Make sure that you, your colleagues and the students have a shared understanding of the verbs that are used.

Select questions according to these criteria:

  • Level of cognitive challenge
  • Syllabus coverage
  • Targeting of key concepts
  • Alignment with learning outcomes

Finally, proofread the exam paper for grammatical and spelling errors.

That’s all, I hope you find it useful!!

Bibliography

Brown, G., Bull, J. & Pendlebury, M. (2003). Assessing student learning in higher education. Routledge.

Brown, S. (2004). Assessment for learning. Learning and Teaching in Higher Education, Issue 1, 2004-05.

Cobb, T. Web Vocabprofile [accessed September 2018 from https://www.lextutor.ca/vp/comp/] an adaptation of Heatley & Nation’s (1994) Range.

Coxhead, A. (2000). A New Academic Word List. TESOL Quarterly, 34(2): 213-238.

Fractus Learning https://www.fractuslearning.com/blooms-taxonomy-verbs-free-chart/

Heatley, A. and Nation, P. (1994). Range. Victoria University of Wellington, NZ. [Computer program, available at http://www.vuw.ac.nz/lals/]

Macquarie University – Faculty of Business and Economics. How to create exams: Learning through assessment. https://www.mq.edu.au/lih/pdfs/FBE_Exams.pdf

Race, P. (2009). Designing assessment to improve physical sciences learning. Higher Education Academy. https://www.heacademy.ac.uk/knowledge-hub/designing-assessment-improve-physical-sciences-learning

10 assessment myths

I believe that the following 10 opinions on assessment and feedback are mistaken. Do you agree?

  1. For the sake of fairness, all students should be assessed identically.
  2. The more summative assessment there is, the greater the impact on learning.
  3. If a higher proportion of students achieve the top grade, standards must be slipping.
  4. Once a marking team has agreed upon essay criteria and standards, consensus will be achieved on grades for individual essays.
  5. There are no valid ways to assess transferable skills such as teamwork or communication.
  6. Formative assessment benefits all students in equal measure.
  7. It is not feasible to measure abstract qualities such as personal integrity or multicultural awareness.
  8. It is best to show students their grades before providing qualitative feedback.
  9. If a student has received quality feedback on a formative assessment task, they will be completely prepared to perform better on a subsequent, related summative assessment.
  10. Setting a summative assessment is the most effective way to motivate learners.

A mini-guide to lesson observations (part 1)

The purpose of this short guide is to help teachers/lecturers and their managers to:

  • be precise about objectives in observing or being observed,
  • follow lesson observation processes accordingly,
  • raise awareness of the skills sets of accomplished observers and observees, and thereby
  • set about participating in lesson observations with increased confidence.

(My apologies if this guide sounds rather dogmatic. I have, however, based it on relevant literature and my experience as a teacher trainer.)

Objectives of lesson observations

  1. Performance appraisal
  2. External inspection
  3. Development of teaching skills
  4. Action research

1. Performance appraisal / 2. External inspection

It is common for teaching staff to be evaluated once or twice a year on their teaching performance through observation. The observer may be a department head or school vice-principal. Some would question the reliability of this form of appraisal, particularly in the case of inspections. See this article by Nick Morrison on Forbes for the arguments against and further references: http://www.forbes.com/sites/nickmorrison/2014/03/19/lesson-observations-are-no-way-to-grade-teachers/

However, for many teaching professionals it is still a reality to be faced. So, it is important to ensure that it is done as carefully as possible. There are many ways to conduct a lesson observation, but I suggest a three-step process based on reflective practice:

Reflection for action (Cowan, 1998)

  • Before the observation

Reflection in action (Schön, 1983)

  • During the observation

Reflection on action (Schön, 1983)

  • After the observation

Before the observation

Ideally, there should be a face-to-face meeting beforehand to clarify several matters. As Brown (1993) said, “It is not enough simply to devise a universal checklist and send line managers out to do it.”

  • The teacher tells the observer about the learners and the lesson objectives
  • The observer takes note of any special considerations
  • The teacher & observer agree on the observation process and focus
  • The teacher & observer make sure they understand the appraisal criteria & standards in the same way. For example, what does “maintains good pace of learning” mean?

During the observation

  • The observer tries not to distract the learners or the teacher
  • The observer pays attention to all criteria that were selected
  • The teacher focuses on the learners and learning and teaches as normally as possible
  • The observer summarises impressions by the end of the lesson

After the observation

  • The teacher completes a self-evaluation before forgetting the details
  • The observer considers the teacher’s self-evaluation and adjusts appraisal or feedback accordingly
  • The observer and teacher exchange views on how well assessment criteria were met, or on the focus of the lesson observation
  • The observer and teacher decide upon future priorities and concrete action points

Professional development

By contrast, this type of lesson observation is characterized by the following terms: voluntary, forward-looking, formative and constructive. Such an observation can be beneficial for lecturers/teachers at any stage of development.

Frequently, the observer is a peer, or possibly an external consultant without vested interests. The role of the observer is to act as a trusted colleague to assist their partner in reflecting on their teaching. The observer does not need to be more knowledgeable about learning-teaching approaches.

I would argue that written impressions by observers should not be recorded and stored in teachers’ records. Otherwise, there is a danger of them being used to inform decisions about contract renewals or promotions.

“The process of observation should be developed between those staff involved.” I suggest that the three-step process above could still be helpful, minus the appraisal element.

The focus of observation is also negotiable. For example, lecturers teaching the same subject could witness alternative ways of presenting the same content, or developing the same skills in students. Alternatively, a lecturer could experiment with a revised learning task and get a second opinion on its design and effec­tiveness. In order to ensure that the observer’s attention remains on the selected focus, an observation instrument can be devised.

Action research

Action research observations are similar in nature to professional development observations in that they are non-evaluative of the lecturer/teacher. They are distinguished by, for instance, their greater formality, use of more precisely designed observation instruments and pre-conceived ways to process collected data. Findings are written up for publication or presentation and may inform revisions in learning/teaching practices or curricula design.

In part two of this mini-guide, I will provide more detail about the skills of participating in lesson observations.

References

Borich, G. D. (1994). Observation skills for effective teaching. New York : Merrill.

Brown, S., Jones, G. & Rawnsley, S. (Eds) Observing Teaching SEDA Paper 79 (Birmingham, Staff and Educational Development Association): 19–22.

Cowan, J. (1998). On becoming an innovative university teacher. Buckingham, UK : SRHE and Open University Press.

Higher Education Academy Resource Pages on Peer Observa­tion of Teaching: http://www.heacademy.ac.uk/resources/detail/subjects/escalate/1043_Peer_Observation_of_Teaching

Montgomery, D. (2002). Helping teachers develop through classroom observation. London : David Fulton.

Schön, D. A. (1983). The reflective practitioner : How professionals think in action. New York : Basic Book.

Key research findings about formative assessment

Formative assessment has a significant impact on learning, as much as 1 or 2 grades on GCSE* results.

Black, P. & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7-74.

(This seminal article was a meta-survey of over 250 publications linking assessment and learning.)

Students can achieve learning goals when they (a) understand the goals, (b) feel like those goals are personal goals, and (c) can evaluate their own progress during courses.

Sadler, D.R. (1989). Formative assessment and the design of instructional systems. Instructional Science 18, 119-144.

Formative assessment promotes effective learning and raises the quality of teaching.

Black. P, Harrison. C, Lee. C, Marshall. B, William, D. (2003). Assessment for Learning: Putting it into practice. Oxford University Press.

Learning gains from formative assessment are disproportionately greater for less-able students.

Assessment Reform Group (2002). Assessment for learning – 10 principles: Research-based principles to guide classroom practice. ARG/Nuffield Foundation.

Formative assessment encourages students to take an active role in the management of their own learning.

Juwah, C. et al. (2004) Enhancing Student Learning Through Effective Feedback. The Higher Education Academy.

Purely formative assessment, with no summative standard measurement, is ineffective.

Smith, E. & Gorard, S. (2005). ‘They don’t give us our marks’: The role of formative feedback in student progress. Assessment in Education, 12(1), 21-38.

*GCSE = General Certificate of Secondary Education in England, Wales & Northern Ireland

The formative and summative assessment disambiguator

Assessment for learning

Assessment of  learning

Also known as…

Formative assessment

Summative assessment

Purposes

To  diagnose student difficulties in reaching learning objectives; To identify student strengths that can be built upon; To inform future learning and teaching.To measure and summarise student achievement of learning objectives in the form of grades; To inform selection procedures for, e.g. promotion, entrance to higher education, employment.

Formality

Formal or informal (ad hoc)

Formal

Timing

Prior to / During school term

During / At end of school term

Consequences

Low-stakes

High-stakes

Perspective

Forward-looking

Backward-looking

Feedback

Specific, qualitative feedback in the form of evaluative comments on performance or   highlighted descriptors on a criteria marking scheme.  Students are provided with constructive and concrete advice on how to improve, time and opportunities for further development and practice. Scores   may be given to show students the result they would have achieved in a ‘real’ test, but the scores do not contribute to grades. Students should absorb teachers’ comments fully before seeing scores.Feedback is optional. If the students are going to do a similar test/exam in future,   then feedback could serve a formative purpose. If the test/exam content is the last time they will be tested on this, then of course no feedback, just grades. If the summative assessment is criterion-referenced, then students can refer to the   criteria marking scheme to understand what the grade implies about their   capabilities, but there is nothing they can do to change their grade.

Feedback providers

Self / Peers / Teacher

Teacher / Examiner

Understanding learning goals

Enhances student understanding of learning objectives, criteria, standards, etc.Merely checks student ability to meet learning objectives

Motivation

Positive

Negative