Exam scheduling: semester or end of year?

Journal Club #6: G. Di Pietro, Bulletin of Economic Research, 2012, 65(1), 65 – 81. [Link]

It is my experience in academic discourse that when a change is proposed, those advocating the change rely on “gut instinct” and “common sense” while those opposing it seek evidence from the literature. My own institution is currently planning a significant change in the academic calendar, and while thinking about this, I came across this paper.

The author examines whether an institution’s reform involving moving semester exams to end of year exams had a negative impact on student performance. The system under study had two semesters, with exams in January and June, and the reform meant that there would be no January exams, just exams for the entire year at the end of the year (the way it used to be, I hear the chorus).

Reasons for the reform included the desire not to overburden first years with exams in January, and to allow students more time to digest their material. The author doesn’t hold back in stating that he believes the reasons for the reform were administrative and financial.

The study involved comparing the mid-term results from modules, and comparing these with semester exam results. Assuming that the mid-term results stayed constant before and after reform, the difference between the mid term mark and the exam performance mark before reform and after reform allow for a measure of the impact of reform on student grades to be determined.

BOER_423_f1

The results shown demonstrate that there was a drop in student performance when the exams moved out of semesters to the end of the year, with students scoring 4 points lower (nearly half a grade).

The author concludes with a statement that sounds a note of caution to those considering changing calendars (DIT colleagues take note!)

These findings may have important policy implications. Changes in examination arrangements should ideally be tested for their impact on student performance before they are introduced. Many changes in higher education are driven not by student learning considerations, but by other reasons such as financial and administrative convenience.

Discussion

Do you have any feelings regarding when modules should be examined?

 

When we grade, do we mean what we say?

The aim of the “Journal Club” is to present a summary of a journal article and discuss it in the comments below or on social meeja. The emphasis is not on discussing the paper itself (e.g. methodology etc) but more what the observations or outcomes reported can tell us about our own practice. Get involved! It’s friendly. Be nice. And if you wish to submit your own summary of an article you like, please do. If you can’t access the paper in question, emailing the corresponding author usually works (This article is available online at the author’s homepage. PDF).

Comments on this article are open until Friday 4th October.
#3: HL Petcovic, H Fynewever, C Henderson, JM Mutambuki, JA Barney, Faculty Grading of Quantitative Problems: A mismatch between values and practice, Research in Science Education, 2013, 43, 437-455.
One of my own lecturers used to remark that he didn’t care how good our workings were, if the final answer was wrong, we’d get no marks. What use is a bridge, he’d ask, if the engineers calculated its span to be too short? We weren’t engineers.

After two weeks of giving out about students, this week I am looking at a paper that probes what lecturers say they expect in student answers, and whether there is a mismatch between these expectations and how they grade. To examine this, the authors have constructed a question with two example answers. The first answer is a detailed, well explained answer that has some errors in it, but these errors cancel out, and give the right answer (ANS A). The second answer is brief, and does not show workings, but gives the correct answer (ANS B). Ten chemistry academics were asked about their grading practices, given these answers, and asked to mark them.

In terms of scoring the solutions, eight of the ten academics scored the incorrect-workings answer (ANS A) higher than the correct-no workings answer (ANS B); and the remaining two scored them equally. The average scores were 7.8 versus 5.1. This was much higher than academics in physics and earth sciences, who were evenly split in whether ANS A scored higher than ANS B.

What do we say we want?

In the interviews, the authors drew up a list of values attributed to instructors in terms of what they wished to see in an answer. Value 1 was that instructors wished to see reasoning in answers to know if the student understands (and to offer specific feedback). All chemistry academics expressed this value.

Value 2 was that instructors wished to find evidence in order to deduct points for incorrect answers. This was interesting, as nine of the ten chemists used this as a reason to deduct points from ANS A, as the student had shown their work; whereas five chemists were reluctant to deduct marks from ANS B as they could not be sure if the student had the same mistakes, as they did not show their workings.

Seven chemists were attributed Value 3, which was a tendency to project correct thinking on ambiguous solutions, assuming that the student writing ANS B must have had the correct thought process, since there was no evidence of a mistake.

Finally, the chemists alone had a fourth value which was not found as much with earth scientists or at all with physicists – a desire to see organisation, units, significant figures; a general methodological approach.

There is evidently a mismatch between the values expressed. Value 1 (want reasoning) and Value 4 (want methodological approach) would appear to conflict with Value 2 (need evidence to deduct) and 3 (projecting correct thought). Most chemists expressed several values, and where they expressed conflicting values, the authors deduced a burden of proof; which set of values the academics (implicitly) rated higher. Six chemists placed the burden of proof on the student: “I can’t see if the student knew how to do this or just copied it.” The remainder placed the burden on themselves: “I don’t want to take any credit off but will tell him directly that he should give more detail.

Message to students

Students of course are likely to take messages from how we grade instead of how we say we will grade. If students are graded with the burden of proof on the instructor, they are more like to do well if they do not expose much reasoning in their answers. If they are required to show reasoning and demonstrate understanding, they are likely to score poorly. Therefore, while we often say that we want to see workings, reasoning, scientific argument, unless we follow through on that, we are rewarding students who call our bluff in this regard!

Discussion

I think this is an interesting paper, and it’s made me think back about how I mark student work. I would imagine that I would be in the burden of proof on the instructor camp, seeing that as implicitly fair to students, but perhaps I need to get a bit harder, and demand fully detailed reasoning in student answers.

  1. Can you identify with any of the four values the authors outline in this paper?
  2. For introductory students, do you have an “induction” of how to illustrate reasoning in answering questions and problems?

Interested to hear what you think.