How to Configure Control Item Settings

About: In SecureMarker, you can use the 'Item Defaults' (per exam) or 'Examiner Competency Settings' (per item) to configure the settings to review examiners and their interactions with Control Items (CIs). This article will detail how to use these settings.

Step 1: Navigate to the 'Quality Control' Screen

Select the 'Quality Control' tab from the SecureMarker interface.

For more information about this screen, see the article 'About the Quality Control Screen'.

Step 2: Select an Exam

First, you must select the appropriate exam to work in from the drop-down menu at the top of the screen.

See the article 'How to Select an Exam to Work In' for more guidance on how to do this.

Step 3: Examiner Competency Settings

If you want to set the competency settings for all items in the exam, select 'Item Defaults'.
If you want to set different competency settings for each item, select the item followed by the 'Examiner Competency Settings' button.

This will display a pop-up where you can configure the settings to review examiners and their interactions with Control Items.

Step 4: Set Daily Competency Rules

These settings are used to determine if an examiner qualifies to mark their quota of items every day.

Option

Description

Number of CIs for daily examiner competency run The number of Control Items that will be presented to an examiner as part of the process of qualifying them to mark their quota of items.
Number of CIs to be marked within tolerance The number of Control Items that must be marked within tolerance in order for the Examiner to qualify for marking their quota of items.
Maximum cumulative error allowed The maximum cumulative tolerance error allowed before an examiner doesn't qualify for marking their quota.

If an examiner marks 5 questions within tolerance, but is 1 mark away (either higher or lower) from the actual mark each time, then they will have a cumulative error of 5. In this case, the examiner will only qualify for marking if the maximum cumulative tolerance is greater than or equal to 5.

Note: If the maximum cumulative error is set to 0, the examiner will be stopped from marking if they deviate from the actual mark for any items.



Step 5: Set Ongoing Marking Rules

Within this group of settings, you are able to define how examiners are continually monitored during 'live' marking, once they have qualified.

Option

Description

Probability of presenting CI The probability of a CI being presented to an Examiner in place of an unmarked response. This setting should be entered as a percentage (100% would mean every item presented is a Control Item, 0% would mean no Control Items would be presented).
Maximum consecutive number of non-CIs that can be presented The maximum number of question item responses that can be presented consecutively, without a Control Item being presented/marked.

In other words, if a Control Item hasn't been presented for X number of items (as set here), then a Control Item will be presented as the next item. This is regardless of any probability calculations.
Maximum consecutive number of CIs that can be marked out of tolerance This sets how many Control Items, in a row, that an examiner can mark out of the set tolerance before being disqualified from continuing their marking activities.

Number of CIs in rolling review A rolling review is an ongoing analysis of an examiner's performance. This setting controls the total number of Control Items marked by an examiner that are used to review examiner performance. For example, if this is set to 10, then the last 10 CIs an examiner marked will be included in the review.
Maximum number of CIs that can be marked out of tolerance in rolling review This is the highest number of Control Items that can be marked out of tolerance in the rolling review of CIs before an examiner is automatically stopped from marking.

Feedback and Knowledge Base