Leveraging Distance-Based Effectiveness Indicators for Enhanced Behavioral Pattern Discovery in Complex Problem-Solving Assessment

Data-driven approaches have emerged as powerful tools for analyzing process data. This study focuses on two data-driven methods: n-gram chi-square feature selection for extracting key action segments and K-medoids clustering combined with Dynamic Time Warping (DTW) distance for identifying behavioral patterns. To address the limitations that arise when applying these methods to complex tasks where ambiguous raw actions often hinder interpretation, this study introduces distance-based effectiveness indicators to enhance both data-driven methods for analyzing actions in the context of complex problem-solving. The research examines how representing action sequences through state effectiveness (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><msub><mrow><mi>d</mi></mrow><mrow><mi>s</mi></mrow></msub></mrow></semantics></math></inline-formula>) and transition effectiveness (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><mi mathvariant="sans-serif">Δ</mi><msub><mrow><mi>d</mi></mrow><mrow><mi>s</mi><mo>→</mo><msup><mrow><mi>s</mi></mrow><mrow><mo>′</mo></mrow></msup></mrow></msub></mrow></semantics></math></inline-formula>) indicators outperforms the use of raw actions alone within the complex collaborative problem-solving Balance Beam task. Results consistently demonstrated that effectiveness indicators significantly improved the sensitivity of n-gram feature selection, the performance of clustering, and the interpretability of both n-grams and resulting clusters. Specifically, state effectiveness representations (<inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><msub><mrow><mi>d</mi></mrow><mrow><mi>s</mi></mrow></msub><mo>→</mo><msub><mrow><mi>d</mi></mrow><mrow><msup><mrow><mi>s</mi></mrow><mrow><mo>′</mo></mrow></msup></mrow></msub></mrow></semantics></math></inline-formula>) yielded the best outcomes. These findings advocate for the integration of effectiveness indicators into data-driven process analytics to more effectively capture and explain behavioral patterns of problem-solving.