How Surpass Calculates P Values, R Values and Discrimination Index (DI) Values

P, R and DI values are item-based statistics; the way they are currently calculated in the Surpass system is demonstrated below.

P Value

The P value represents the mean mark of an item, scaled to a value between 0 and 1.

To complete this calculation, the total number of values (the amount of the times the question was attempted) is multiplied by the total marks available for the item in question and the sum of this is divided by one.  This amount is then multiplied by the sum of all the marks for the total amount of candidates who answered the question.  

An example of this would be, if an item was worth 5 marks, 10 candidates attempted it and their marks were:    
 
2, 5, 4, 1, 2, 0, 5, 5, 3, 3  

The item’s P value would be:                  

(1 / (10 * 5)) * (2 + 5 + 4 + 1 + 2 + 0 + 5 + 5 + 3 + 3)   =            

0.02 * 30 =            

0.6

Note: If items have been migrated to Surpass with an existing P Value, the P Value will be updated from the data already present and will not be generated from scratch. This is so that users have an accurate P Value for all items.

 
R Value

 

Where:

  • x is the score on the first variable, which in the case of the Item Performance Report statistics will be the task score (e.g. score of 0-5 for Task 1 in the reading test in Table 2).
  • y is the score on the second variable, which in the case of the Item Performance Report Statistics will the total test score.
  • x ̅  is the mean for variable 1, which in the case of the Item Performance Report will be the mean of task scores (x).
  • y ̅  is the mean for variable 2, which in the case of the Item Performance Report will be the mean of total test score (y).



Discrimination Index (DI)


The Discrimination Index in Reporting is calculated by ordering all the candidate scripts (descending) based on the total script mark and then splitting them up into three equal groups. The top group will consist of all the candidates whose script marks were in the top third (the highest achievers) and the bottom group will consist of all the candidates whose script marks were in the bottom third (the lowest achievers). If the groups do not divide exactly in to three then the remainder of candidates (up to 2) are placed in to the middle third to ensure that the top and bottom groups are of equal size; the middle group has no further place in the calculation.  


The top group and the bottom group of candidates are then compared on average for each item in the test.  This is done by calculating the P value for each item for both the top group and the bottom group and then subtracting the bottom group’s P value from the top group’s P value.  



For example, if the P value for the top group of Item A was 0.8 and the P value for the bottom group was 0.2 then the DI would be 0.8 – 0.2 = 0.6. This calculation is then done for each item in the test.

  • If the top group did better on average then the DI will be positive (between 0 and 1)
  • If the bottom group did better on average then the DI will be negative (between 0 and -1) 
  • If the two groups did exactly the same the DI will be 0   
  • If all the top group answered a question item correctly and all the bottom group answered it incorrectly then the DI will be 1
  • If all the bottom group answered it correctly and all the top group answered it incorrectly then the DI will be -1

 The best circumstances to calculate the DI value is with a group of candidates who have all attempted the same items; the calculation is most effective for large cohorts taking fixed tests.

For more information about reporting, visit the Reporting section of the Surpass Help. This includes informations about:
Operational Data
Test Forms Report
Item Bank (Item Production) Report
Items Delivered Report
Tests Delivered Report


Feedback and Knowledge Base