[PDF] Making Sense of Item Response Theory in Machine Learning | Semantic Scholar (2024)

A new IRT model, particularly designed for dealing with nonnegative unbounded responses, which is adequate for modelling the absolute errors of regression algorithms is proposed.

The proposed β-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves, outperforms a more standard 2PL-ND model on all datasets and is applied to assess the ability of machine learning classifiers.

The proposed β³-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Curves, outperforms a more standard 2PL-ND model on all datasets and is applied to assess the ability of machine learning classifiers.

A novel probabilistic framework to improve the accuracy of a weighted majority voting algorithm by introducing the item response theory (IRT) framework to evaluate the samples’ difficulty and classifiers’ ability simultaneously.

Comparisons with the most popular filter‐based FS methods show that it is possible to obtain better results with this new modified selector or one of its variants on the majority of both binary and real‐world datasets compared to its well‐known peers.

This work proposes a new assessment methodology based on the combination of Item Response Theory (IRT) and Glicko-2, a rating system mechanism generally adopted to assess the strength of players, which identified the Random Forest as the algorithm with the best innate ability.

Adaptive approaches to achieve better efficiency on model benchmarking are investigated, adapting existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing.

...

...

A case study in which instance hardness is measured by fitting the responses of Random Forests with different number of trees is developed, which reveals several insights about different levels of discrimination among instances, the adequate number of Trees in RF and anomalous situations that were related to noisy instances.

This work proposes the use of Thornton’s Separabilit y Index as a simple measure of subset merit which is fast and easy to calculate, but gives results which are identical to the asymptotic result of multiple testing with random data splits.

This paper identifies instances that are hard to classify correctly (instance hardness) by classifying over 190,000 instances from 64 data sets with 9 learning algorithms and finds that class overlap is a principal contributor to instance hardness.

The Rasch Models for Ordered Polytomous Data and the Generalized Partial Credit Model: Conceptual Development of the Multiple-Choice Model, and Issues to Consider in Selecting among the 1PL, 2PL, and 3PL Models.

This book discusses several approaches to obtaining knowledge concerning the performance of machine learning and data mining algorithms and shows how this knowledge can be reused to select, combine, compose and adapt both algorithms and models to yield faster, more effective solutions to data mining problems.

Chris Thornton makes the compelling claim that learning is not a passive discovery operation but an active process involving creativity on the part of the learner, in which the results of one learning step serve as the basis for the next.

...

...