Broken Assessment: Teaching to Unfair Tests

Broken Assessment: Teaching to Unfair Tests

#BrokenAssessment

“This mark scheme encourages learning the syllabus off-by-heart and nothing more.”

Dr. Helen Pennington

“If a board certified Stanford trained Ob/Gyn physician could only get 1 point on this question, I don’t expect that high school students could do any better.”

Dr. Ruth Ann Crystal

Summary: Until recently I’ve come to accept that training students to interpret subtle nuances in assessment questions is acceptable and necessary practice. But the truth is I’m here to teach Science, not “exam skills”. I asked a few experts to answer some IGCSE Biology questions, on the assumption that they should, if the assessment is fair and appropriate, be able to get full marks, given their superior level of knowledge and skills. They didn’t. 

Here I highlight some of the problems in the way we assess students in high school. Is high school assessment broken? Watch the video/read the blog and decide for yourself. Reach out to me on Twitter (@science_sauce) and share your opinions. #BrokenAssessment

Is high school assessment broken?

For test review in class I typically give student’s their marked papers back, display the mark scheme on the big screen, and go over each question one at a time. I’m sure this is standard practice for this types of assessment.

I should point out that I’m referring here to externally created examination papers. Any unit/chapter/topic tests I do in class are taken from past papers, and aren’t written by me.

It’s a common theme that, when looking at the mark scheme, students say something like, “I know all the things in that mark scheme, I just didn’t know the question was asking me to say those things.” It’s a classic problem; interpreting questions and deciding what they require is a challenge for students. But why should it be? Why have we come to accept that as the norm? Why is it acceptable to us that we have to spend so much of our courses dedicated to preparing students for how to interpret a test, and not on teaching them skills and knowledge related to our subject?

A month or two ago I was marking a paper and noticed a few things in the mark scheme that I didn’t think were fair: an instance where the the mark scheme required some statements that I didn’t feel the question clearly asked for, an instance where something I’m confident is correct wasn’t accepted etc. And since this isn’t the first time I’ve been in this situation, I decided to take it a little further…

My course should provide students (16 year old students, in the case of these exams) with a certain level of knowledge and skill in Biology, and the exams should assess that knowledge. An expert in the field, e.g. a professional in that subject area or someone holding a Ph.D, should logically have knowledge and skills way beyond what is expected of my 16 year old students and so should surely get full marks.

If a person can’t get full marks there are, in my opinion, two possibilities:

  1. The person doesn’t have enough knowledge/skills in the subject area.
  2. The question isn’t effectively assessing the knowledge and skills the person has.

By involving experts in the field, I can safely rule out possibility number one.

So, I put it to the test and found some experts to answer a few questions. I put a few shoutouts on Twitter and some people got back to me.

Plant Biologists: Dr. Steve Cook and Dr. Helen Pennington

Photosynthesis 2 of2
(From a previous question, P is identified as carbon dioxide, and Q is identified as plants.)

Dr. Cook and Dr. Pennington each have a Ph.D in a different area of Plant Biology and both have a very strong knowledge of photosynthesis. I presented them with a five mark question that requires students to describe the process of photosynthesis and each of them (both experts, remember) got four marks out of five.

photosynth answers
The mark scheme for the question in photosynthesis. Each marking point (1-7) is worth one mark, with a maximum of 5 marks for this question.

They each stated the same four marking points (3, 4, 5 and 6) but missed off marking points 1, 2 and 7. The question asks how plants convert carbon dioxide to carbon compounds in plants, but it doesn’t ask how the carbon dioxide enters the leaves and cells, yet marking points 1 and 2 involve that. Marking point 7 requires students to state the balanced chemical equation. This is not explicitly required in the question.

“If I were writing a short-answer question like this, I would be clearer that this is what I expected.”

Dr. Cook is a Principal Teaching Fellow (and also the holder of a PGCE) at London Imperial. He says, “If I were writing a short-answer question like this, I would be clearer that this is what I expected.”

I shared my opinion with Dr. Cook that marking point 7 (the balanced equation) isn’t required by the question and he states, “Completely agree that the equation bit is nonsense: if that is what they wanted (e.g. because an ILO is “be able to write a balanced equation”), then they should have asked for that explicitly in the question text.”

I asked Dr. Pennington her thoughts on the mark scheme: “A student would be able to get high or full marks by mentioning the correct buzzwords without any real context.” Says Dr. Pennington. She goes on to state, “This mark scheme encourages learning the syllabus off-by-heart and nothing more.”

Nutritionist: Megan Pfeffer

Oct Nov 2013 Q

I spoke to Megan Pfeffer, a degree-qualified clinical nutritionist and founder of I Choose Health, a classic IGCSE question: “Explain the term balanced diet.” For this three mark question Megan scored zero, as she mentioned none of the points in the mark scheme. She stated in her answer that a balanced diet “is one that takes into consideration health status, age, race, culture… [and other factors]”, but she didn’t define “diet”.

oct nove 2013 MS

When I discussed her answer and the mark scheme she said, “The word ‘diet’ implies molecules for metabolism… ‘Balanced’ is the word seeking definition in the question, not, “what does ‘diet’ mean” .”

I recognise that this is merely an interpretation of the question, but that fully demonstrates my point. Her interpretation was valid and her answer based on that interpretation was also valid. If a qualified nutritionist interpreted the question like that, is it fair to penalise students who do the same?

Perhaps the question should read, “Define the term diet and explain what it means for a diet to be balanced.” This would perhaps take some of the mystery out of the process.

OB/GYN Physicians: Dr. Leslea Walters and Dr. Ruth Ann Crystal

Lastly, I took a simple question on pregnancy, and asked a few medical practitioners to respond with what they thought were appropriate answers. The question reads, “Explain why it is especially important for pregnant women to include milk and dairy products in their diet.”

pregnancy q

Dr. Leslea Walters scored two out of two by mentioning calcium being needed for bones.

Screen Shot 2019-02-10 at 19.35.27

Dr. Walters highlighted a major weakness in the question: that dairy products do no have to be consumed to provide calcium. She comments, “That is a question made based on an answer. But the answer to the question itself is much more broad.” She goes on to say, “A student whose knowledge is limited to what was taught may find the answer to be obvious, but a student with broader knowledge could be misled and lose points for knowing more!”

Dr. Ruth Ann Crystal scored only one point out of two with the answer, “…for the calcium (and protein) in the diet because the fetus will take the woman’s calcium.” To break her statement down, I feel she’s stated the need for dairy (it contains calcium), and explained it (the fetus will take it from the mother). These are two appropriate marking points, in my opinion, making a suitable two mark answer to the question. However, the absence of any mention of “bones” or “skeleton” means she can’t gain the second mark.

“It’s not a good question. It is a “read my mind” question.”

Dr. Crystal states, “I would say it is not a very well worded question. If they want you to talk about the bones, they should ask what it could affect in the mother which would be loss of calcium in the bones (and possibly the teeth)… “It’s not a good question. It is a “read my mind” question.”

As if reading my mind, Dr. Crystal stated, “If a board certified Stanford trained Ob/Gyn physician could only get 1 point on this question, I don’t expect that high school students could do any better.”

Nail on the head, Doctor, nail on the head.

What are your experiences?

Share your thoughts. Are you getting frustrated with the same problems in assessment? How is it affecting your students? Reach out to me and Twitter (@science_sauce) and share your opinions. #BrokenAssessment

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: