Revealing Rubric Relations: Investigating the Interdependence of a Research-Informed and a Machine Learning-Based Rubric in Assessing Student Reasoning in Chemistry

Abstract

Valid rubrics facilitate assessing the level of complexity in students’ open-ended responses. To design a valid rubric, it is essential to thoroughly define the types of responses that represent evidence of varying complexity levels. Formulating such evidence statements can be approached deductively by adopting predefined criteria from the research literature or inductively by detecting topics, for example, based on data-driven machine learning (ML) techniques. Investigating the interdependence of such research-informed and ML-based rubrics is key to validating ML-based approaches and enhancing their applicability in formative assessments. This study quantitatively compares a research-informed and an ML-based rubric designed to capture the complexity of students’ reasoning on the relative rate of contrasted reactions in undergraduate organic chemistry. Specifically, we leveraged an ML-based clustering technique to inductively develop a holistic fifteen-category rubric to evaluate students’ open-ended reasoning. Subsequently, we performed a quantitative analysis to examine whether the ML-based rubric and its research-informed counterpart are significantly associated. Our findings indicate that research-informed and ML-based rubrics assess students’ reasoning comparably. Thus, both rubric types are valid for categorizing students’ reasoning, underscoring the applicability of integrating ML techniques into rubric development. Nevertheless, aligning ML-based rubrics with the respective assessment objectives remains crucial. A well-aligned, evidence-based rubric may ultimately ease the analysis of student reasoning.