Item response theory in AI: Analysing machine learning classifiers at the instance level | Semantic Scholar (2024)

A new IRT model, particularly designed for dealing with nonnegative unbounded responses, which is adequate for modelling the absolute errors of regression algorithms is proposed.

This research explores the Item Response Theory (IRT) as a tool to explaining the models and measuring the level of reliability of the Explanation-by-Example approach.

This paper introduces the concept of an assessor model, \hat{R}(r|\pi,\mu), a conditional probability estimator trained on test data, and proposes accompanying every deployed AI system with its own assessor.

A novel probabilistic framework to improve the accuracy of a weighted majority voting algorithm by introducing the item response theory (IRT) framework to evaluate the samples’ difficulty and classifiers’ ability simultaneously.

Comparisons with the most popular filter‐based FS methods show that it is possible to obtain better results with this new modified selector or one of its variants on the majority of both binary and real‐world datasets compared to its well‐known peers.

The robustness of different families of machine learning models are evaluated, which are selected and characterised according to their behaviour and a novel taxonomy is defined based on the robusts of the different models and the difficulty of the instances addressed.

This demonstration shows that using the item response theory (IRT) to stratify the data based on how difficult each case is to classify, independent of the outcome measure of interest (eg, accuracy), and highlights how well classifiers differentiate cases of varying difficulty.

This study aims to demonstrate how the item response theory can be used to stratify the data based on how difficult each case is to classify, independent of the outcome measure of interest (eg, accuracy), and highlights how well classifiers differentiate cases of varying difficulty.

A new evaluation methodology based on IRT as well as the tool decodIRT, developed to guide IRT estimation over ML benchmarks are presented.

This work proposes a new assessment methodology based on the combination of Item Response Theory (IRT) and Glicko-2, a rating system mechanism generally adopted to assess the strength of players, which identified the Random Forest as the algorithm with the best innate ability.

...

...

In this paper, a series of experiments with a range of datasets and classification methods are performed to fully understand how IRT works and what their parameters really mean in the context of machine learning.

A case study in which instance hardness is measured by fitting the responses of Random Forests with different number of trees is developed, which reveals several insights about different levels of discrimination among instances, the adequate number of Trees in RF and anomalous situations that were related to noisy instances.

This paper investigates how training size and the incorporation of noise affect a DNN's ability to generalize and learn, and finds that different DNN models exhibit different strengths in learning and are robust to noise in training data.

This paper identifies instances that are hard to classify correctly (instance hardness) by classifying over 190,000 instances from 64 data sets with 9 learning algorithms and finds that class overlap is a principal contributor to instance hardness.

The ISMB algorithm was evaluated on 21 UCI data sets using five different performance measures and its performance was compared to that of several commonly used predictive algorithms, including nave Bayes, C4.5 decision tree, logistic regression, neural networks, k-Nearest Neighbor, Lazy Bayesian Rules, and AdaBoost.

This paper critically assess the different ways AI systems are evaluated, and the role of components and techniques in these systems, and identifies three kinds of evaluation: human discrimination, problem benchmarks and peer confrontation.

This work proposes the use of Thornton’s Separabilit y Index as a simple measure of subset merit which is fast and easy to calculate, but gives results which are identical to the asymptotic result of multiple testing with random data splits.

This work is focused on presenting a survey of the main instance selection methods reported in the literature, and shows how the training set is reduced which allows reducing runtimes in the classification and/or training stages of classifiers.

...

...