MEASURING THE PERFORMANCE OF ORDINAL CLASSIFICATION
Abstract
Ordinal classification is a form of multiclass classification for which there is an inherent order between the classes, but not a meaningful numeric difference between them. The performance of such classifiers is usually assessed by measures appropriate for nominal classes or for regression. Unfortunately, these do not account for the true dimension of the error.
The goal of this work is to show that existing measures for evaluating ordinal classification models suffer from a number of important shortcomings. For this reason, we propose an alternative measure defined directly in the confusion matrix. An error coefficient appropriate for ordinal data should capture how much the result diverges from the ideal prediction and how "inconsistent" the classifier is in regard to the relative order of the classes. The proposed coefficient results from the observation that the performance yielded by the Misclassification Error Rate coefficient is the benefit of the path along the diagonal of the confusion matrix. We carry out an experimental study which confirms the usefulness of the novel metric.
References
S. Baccianella , A. Esuli and F. Sebastiani , Evaluation measures for ordinal regression, Proc. Ninth Int. Conf. Intelligent Systems Design and Applications (2009) pp. 283–287. Google Scholar- Eng. Appl. AI 20(7), 875 (2007). Crossref, Google Scholar
- Artif. Intell. Med. 40, 115 (2007), DOI: 10.1016/j.artmed.2007.02.007. Crossref, Web of Science, Google Scholar
- J. Mach. Learn. Res. 8, 1393 (2007). Web of Science, Google Scholar
- C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, 2001, Software available at http://www.csie.ntu.edu.tw/∼cjlin/libsvm . Google Scholar
E. Frank and M. Hall , A simple approach to ordinal classification, Proc. 12th European Conf. Machine Learning (Springer-Verlag, 2001) pp. 145–156. Google ScholarL. Gaudette and N. Japkowicz , Evaluation methods for ordinal classification in, Proc. 2nd Canadian Conf. Artificial Intelligence (Springer, 2009) pp. 207–210. Google ScholarT. J. Hastie and R. J. Tibshirani , Generalized Additive Models,Monographs on Statistics and Applied Probability 43 (1990) pp. 297–318. Google ScholarR. Herbrich , T. Graepel and K. Obermayer , Support vector learning for ordinal regression, Ninth Int. Conf. Artificial Neural Networks ICANN1 (1999) pp. 97–102. Google Scholar- Int. J. Patt. Recogn. Artif. Intell. 6, 1121 (2008). Google Scholar
- Biometrika 30, 81 (1938). Crossref, Google Scholar
- Int. J. Artif. Intell. Soft Comput. 2, 46 (2010), DOI: 10.1504/IJAISC.2010.032512. Crossref, Google Scholar
- Proc. Int. Conf. Machine Learning and Cybernetics 4, 2220 (2002), DOI: 10.1109/ICMLC.2002.1175434. Crossref, Google Scholar
- J. Roy. Stat. Soc. Series B 42, 109 (1980). Google Scholar
- Neural Networks 21, 78 (2008), DOI: 10.1016/j.neunet.2007.10.003. Crossref, Web of Science, Google Scholar
-
J. Pinto da Costa , R. Sousa and J. S. Cardoso , An all-at-once unimodal SVM approach for ordinal classification , Proc. Ninth Int. Conf. Machine Learning and Applications (ICMLA 2010) ( 2010 ) . Google Scholar -
W. Press , Numerical Recipes in C: The Art of Scientific Computing ( Cambridge University Press , Cambridge , 2002 ) . Google Scholar - IEEE Trans. Patt. Anal. Mach. Intell. 30(7), 1158 (2008), DOI: 10.1109/TPAMI.2007.70776. Crossref, Web of Science, Google Scholar
- , Adv. Neural Inform. Process. Syst 15, eds.
Thrun and K. Obermayer (MIT Press, 2003) pp. 937–944. Google Scholar - American J. Psychol. 15, 72 (1904), DOI: 10.2307/1412159. Crossref, Web of Science, Google Scholar
- Biometrics 59, 263 (2003), DOI: 10.1111/1541-0420.00033. Crossref, Web of Science, Google Scholar
- Statist. Methodol. 6(2), 157 (2009), DOI: 10.1016/j.stamet.2008.06.001. Crossref, Google Scholar
- W. Waegeman, Learning to Rank: a ROC-Based Graph-Theoretic Approach, Ph.D. thesis, Universiteit Gent, 2009 . Google Scholar
-
W. Waegeman , B. De Baets and L. Boullart , A comparison of different ROC measures for ordinal regression , Proc. CML 2006 Workshop on ROC Analysis in Machine Learning ( 2006 ) . Google Scholar
