TitlePercentile analysis for goodness-of-fit comparisons of models to data
Publication TypeConference Proceedings
Year of Conference2014
AuthorsKhemlani, SS, Trafton, JG
Conference NameProceedings of the 36th Annual Conference of the Cognitive Science Society
Pagination737-742
Date Published07/2014
PublisherCognitive Science Society
Conference LocationQuebec City, Canada
Abstract

In cognitive modeling, it is routine to report a goodness-of-fit index (e.g., R² or RMSE) between a putative model's predictions and an observed dataset. However, there exist no standard index values for what counts as “good” or “bad”, and most indices do not take into account the number of data points in an observed dataset. These limitations impair the interpretability of goodness-of-fit indices. We propose a generalized methodology, percentile analysis, which contextualizes goodness-of-fit measures in terms of performance that can be achieved by chance alone. A series of Monte Carlo simulations showed that the indices of randomized models systematically decrease as the number of data points to be fit increases, and that the relationship is nonlinear. We discuss the results of the simulation and how computational cognitive modelers can use them to place commonly used fit indices in context.

Full Text
pdf: 
https://www.nrl.navy.mil/itd/aic/sites/www.nrl.navy.mil.itd.aic/files/pdfs/khemlani%20cog%20sci%202014%202.pdf
NRL Publication Release Number: 
14-1231-0478