Standard setting simplified - Cohen

What is Cohen?

Cohen_method_standard_setting_overview

Cohen is a method of establishing a cut-off score by taking 60% of the score achieved by the top performing candidate (usually the 95th percentile). This simple and inexpensive method is described as a compromise method, as it has elements of both absolute methods (based on the performance of candidates in relation to a defined standard) and relative methods (where the number of passing candidates is relative to the rest of the candidates taking the exam).

Why 60%?

The original Cohen method was proposed by Cohen-Schotanus and Van der Vleuten in 2010. They suggested that using 60% of the top performing candidates score would be more effective than using an arbitrary percentage (most commonly, 60% is used as the cut-off score in Europe) as the top performing student is the best representation of the highest score that could be achieved on an exam. Read more about their research here.

The Cohen method is considered less valid than certain other methods of standard setting as the only thing that separates it from an arbitrary cut-off score (for example, one standard error of measure below 50%) is that it bases the cut-off score on the performance of the highest performing candidate. This has more credibility than an arbitrary method as it uses the highest performing students score as a stable estimate of the best result that can be achieved on this test.  This is debatable though as it assumes that the top performing student is an accurate representation of the rest of the students’ ability and also that the highest performer is consistently this way.

How is Cohen calculated?

Cohen is calculated by taking the candidate in the 95th percentile (the candidate whose score is higher than 95% of the rest of the candidates taking the same exam) and finding 60% of their score. This is considered the cut-off score that a candidate would be expected to achieve to pass the exam.  For example, if the total mark of the exam is 120 and the 95th percentile student got 92 marks, then the Cohen cut-off score would be 55/120 (46%) – which is 60% of 92.

students_learning_cohen

Should you use Cohen?

The Cohen method is very affordable and simple to apply. However, questions can be raised about the validity of the method as there is no real justification or evidence behind using 60% of the 95th percentile.  However, it does have some substance in that the top performing student is a good representation of the best that could be achieved in the exam, and this is also a stable figure. For these reasons, the Cohen method is ideal for use in in-house, informal examinations but not for high-stakes examinations that have implications on people’s careers.

Advantages of Cohen

Quick & cheap – Cohen is a very quick method of calculating a cut-off score and doesn’t require a panel of judges which can be difficult and expensive to arrange, making it ideal for informal in-house exams.

Simple – The maths behind the Cohen method is simple, you simply need to calculate 60% of the result of the 95th percentile student, and make amendments for guessing.

Reduced variability – Studies have found that Cohen consistently produces similar cut-off scores for the same exam over different years and cohorts of students.

Disadvantages of Cohen

Not ideal for high stakes - This method is not very valid or reliable for use in high stakes examinations that decide the fate of student’s careers or progression in education as 60% could be considered an arbitrary measurement.

Assumptions – This method assumes that the student in the 95th percentile is a consistent high performer, and also that they are a fair representation of the abilities of the rest of the cohort.

In summary, the Cohen method is a quick and simple method of calculating a cut-off score in low stakes, informal examinations. It would be advised to consider using a more justifiable method of standard setting such as Angoff or Borderline regression for high stakes examinations.

Coming soon…

Look out for the next blog in our “Standard setting simplified” series... Ebel.