NOTE: Only Site Owners, selected staff, and KU staff can see the Secondary Scoring section. Go to Other Permission Roles to see how to grant permission to staff who need to do scoring but do not need full control of the site.
The Secondary Scoring feature of the Community Check Box (CCB) is designed to serve as a user-friendly platform for assuring the quality and accuracy of the data in Accomplishments entries. The scoring includes entering secondary scores, calculating rates of inter-observer reliability, viewing coding discrepancies, providing comments and feedback on documented events, reflecting on next steps for improving levels of inter-observer agreement, generating recommendations for improvement, and sharing overall feedback and ideas for improvement with those entering data.
Customized Secondary Scoring Sheets can be created for scoring and providing feedback on data (for example, on a monthly or quarterly basis). You set the parameters for which data you would like to review, and once Scoring Sheets are created, you can save and resume your work at any time. Data entered in Secondary Scoring is synchronized with CCB Accomplishments, so you can select to display a green or red flag next to that entry in the Accomplishments listing. See Secondary Scoring Flags section below for details.
We will use the terms Observer 1, Observer 2, and Observer 3 to refer to the following:
CCB Administrators assign visibility and access to Secondary Scoring for users based on their specific needs and responsibilities. Secondary Scoring can be accessed from the left navigation bar in the CCB. From this Secondary Scoring Platform, secondary and tertiary scorers can easily access the data they need to review.
No Flag – Indicates the record has not yet been scored by Observer 2 or Observer 3Green Flag – Indicates agreement on scoring by all Observers who have coded the entryYellow Flag - Indicates lack of agreement on scoring between Observer 2 and Observer 3Red Flag – Indicates lack of agreement on scoring between Observer 1 and Observer 2
No Flag – Indicates the record has not yet been scored by Observer 2 or Observer 3
Green Flag – Indicates agreement on scoring by all Observers who have coded the entry
Yellow Flag - Indicates lack of agreement on scoring between Observer 2 and Observer 3
Red Flag – Indicates lack of agreement on scoring between Observer 1 and Observer 2
Secondary Scoring is an optional section available to all projects using the CCB. By default, the section is not displayed until a determination has been made if it will be used by a project. Administrators need to determine who within the project will serve as Observer 2 and Observer 3 and provide access to the Secondary Scoring section. Documenters do not need access to this tool.
Administrators also need to determine if flags should be displayed within Accomplishments to identify if there is Observer 1 and Observer 2 agreement on entries. If selected, display of the flags in the Accomplishments is a feature that needs to be enabled. As Administrator, you can select whether your site users will be able to view flags which link to the related Secondary Scoring feedback.
Before you can use Secondary Scoring, you need to turn it on for the CCB form you will score.
NOTE: When you first click on Secondary Scoring, the only action link that will be available is New. After you select a previously created scoresheet, the other options of Edit, Delete, Scoring and Download will be active.
To complete a score sheet, follow protocols outlined for your project to score an entry.
If a flag is present in the score sheet, this indicates the entry has previously been scored by another observer. You can change the score selected, or leave as-is.
Once secondary scoring is complete, you can generate a report to reflect on results and provide to users documenting their entries within CCB. To download the scoring report:
The Secondary Scoring Report will consist of a Cover Page, Score Sheet Data, and at least one Agreement Summary, each available on a separate tab at the bottom of the MS Excel worksheet. Take a look at this example:SecondaryScoringReport Example.xlsx.
The Cover Page serves as a quick reference to the overall outcome of the secondary scoring. It features the Score Sheet Title, Observer Name, Date Range, Number of Entries and % Agreement (between Observer 1 and Observer 2). You can customize this page in Excel to include additional details as appropriate for the project.
Agreement Summary displays the inter-reliability report of each Observer's scores. This report includes:
A table will outline the agreement summary and highlight the agreement between observers.
If a secondary score has been created for an entry, and if flags have been turned on for this CCB form, a flag will appear on the Accomplishments View Data page to indicate whether or not there is a match between the code provided by Observer 1 and Observer 2.
To access the screen shown below on your WorkStation: