Beyond a snapshot: Using machine learning to monitor and adaptively support organic chemistry students’ mechanistic reasoning over time

Abstract

Science education seeks to equip students with the skills to engage with the natural and engineered world. Mechanistic reasoning is essential for achieving this goal because it enables students to identify, describe, or analyze the underlying mechanisms of observed phenomena to explain, predict, or control their outcomes. However, students often use rote memorization or pattern recognition when identifying mechanisms, rather than engaging in mechanistic reasoning. As a result, they may redescribe mechanisms without identifying cause-and-effect relationships or view mechanisms as linear sequences of unrelated steps. To address this, explicitly supporting students’ mechanistic reasoning in science instruction is essential. Nonetheless, research shows that uniform support is less effective in guiding students’ mechanistic reasoning than adaptive support. Yet, adaptively supporting students’ mechanistic reasoning is challenging due to the time-consuming process of manual coding. Furthermore, our understanding of how mechanistic reasoning develops over time and interacts with other cognitive and affective variables remains limited, complicating the design of adaptive support. This dissertation addresses these challenges from a methodological and pedagogical perspective. From a methodological perspective, it explores the potential of unsupervised machine learning (ML) to capture the heterogeneity of undergraduate organic chemistry students’ mechanistic reasoning about alternative reaction products. It also examines how supervised ML can automatically categorize this heterogeneity over time using a fine-grained scoring rubric. From a pedagogical perspective, this dissertation investigates how students integrate essential epistemic heuristics for mechanistic reasoning—operationalized through their use of granularity, disciplinary core ideas, and causality—using quantitative methods. In detail, it explores how these heuristics evolve, how they relate to other cognitive and affective variables, and how they are influenced by adaptive support. Our findings indicate that unsupervised ML techniques can detect significant heterogeneity in how students demonstrate mechanistic reasoning. Specifically, our analysis reveals that students integrate diverse disciplinary core ideas, interconnected at varying levels of granularity and causality, into their mechanistic reasoning. The extent to which students integrate these heuristics depends on the specific task affordances. Building on these findings, we developed a supervised ML model that automatically evaluates students’ mechanistic reasoning in line with these heuristics based on a fine-grained 24-category rubric. This automated scoring model allowed us to monitor and adaptively support students’ mechanistic reasoning beyond snapshots in time. A quantitative analysis of the collected data revealed a statistically significant relationship between students’ use of granularity and causality. Integrating disciplinary core ideas related to energy contributed to more complex mechanistic reasoning, along with high chemical concept knowledge and confidence when solving organic chemistry exercises. However, while many students improved their mechanistic reasoning due to the adaptive support over time, not all students benefited equally. Overall, this dissertation contributes to the existing literature by advancing the methodological rigor and pedagogical utility of ML in monitoring and supporting students’ mechanistic reasoning in undergraduate science courses. By combining the strengths of human and machine analysis, it demonstrates how to thoughtfully integrate ML methods into research processes to generate evidence-based insights into students’ mechanistic reasoning. Leveraging ML facilitated large-scale analysis of students’ mechanistic reasoning over time, guiding the design of adaptive support tailored to their mechanistic reasoning skills.