Regular PaperNo Access

A Generalized Evaluation Framework for Multimedia Recommender Systems

Mouzhi Ge

Faculty of Informatics, Masaryk University, Brno 60200, Czech Republic

E-mail Address: [email protected]

Search for more papers by this author

and

Fabio Persia

Faculty of Computer Science, Free University of Bozen-Bolzano, Bozen-Bolzano 39100, Italy

E-mail Address: [email protected]

Search for more papers by this author

https://doi.org/10.1142/S1793351X18500046Cited by:5 (Source: Crossref)

Abstract

With the widespread availability of media technologies, such as real-time streaming, new Internet-of-Thing devices and smart phones, multimedia data are extensively increased and the big multimedia data rapidly spread over various social networks. This has created complexity and information overload for users to choose the suitable multimedia objects. Thus, different multimedia recommender systems have been emerging to help users find the useful multimedia objects that are possibly preferred by the user. However, the evaluation of these multimedia recommender systems is still in an ad-hoc stage. Given the distinct features of multimedia objects, the evaluation criteria adopted from the general recommender systems might not be effectively used to evaluate multimedia recommendations. In this paper, we therefore review and analyze the evaluation criteria that have been used in the previous multimedia recommender system papers. Based on the review, we propose a generalized evaluation framework to guide the researchers and practitioners to perform evaluations, especially user-centric evaluations, for multimedia recommender systems.

Keywords:

References

1. G. Adomavicius and A. Tuzhilin , Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions, IEEE Trans. Knowl. Data Eng. 17 (6) (2005) 734–749. Crossref, Web of Science, Google Scholar
2. M. Albanese, A. d’Acierno, V. Moscato, F. Persia and A. Picariello , A multimedia recommender system, ACM Trans. Internet Technol. 13 (1) (2013) 32. Crossref, Web of Science, Google Scholar
3. M. F. Alhamid, M. Rawashdeh, H. Al Osman, M. S. Hossain and A. El Saddik , Towards context-sensitive collaborative media recommender system, Multimedia Tools Appl. 74 (2015) 11399–11428. Crossref, Web of Science, Google Scholar
4. B. Al Takrouri, K. Detken, C. Martinez, M. K. Oja, S. Stein, L. Zhu and A. Schrader , Mobile holstentour: Contextualized multimedia museum guide, in Proc. 6th Int. Conf. Advances in Mobile Computing and Multimedia, 2008, pp. 460–463. Google Scholar
5. F. Amato, F. Gargiulo, V. Moscato, F. Persia and A. Picariello , Recommendation of multimedia objects for social network applications, in Proc. Workshops of the EDBT/ICDT 2014 Joint Conf., 2014, pp. 288–293. Google Scholar
6. F. Amato, V. Moscato, A. Picariello and F. Piccialli , Sos: A multimedia recommender system for online social networks, Future Gener. Comput. Syst. (2017), in press. Crossref, Web of Science, Google Scholar
7. S. S. Anand, P. Kearney and M. Shapcott , Generating semantically enriched user profiles for web personalization, ACM Trans. Internet Technol. 7 (4) (2007) 22. Crossref, Web of Science, Google Scholar
8. I. Bartolini, Z. Zhang and D. Papadias , Collaborative filtering with personalized skylines, IEEE Trans. Knowl. Data Eng. 23 (2) (2011) 190–203. Crossref, Web of Science, Google Scholar
9. I. Bartolini, V. Moscato, R. G. Pensa, A. Penta, A. Picariello, C. Sansone and M. L. Sapino , Recommending multimedia visiting paths in cultural heritage applications, Multimedia Tools Appl. 75 (7) (2016) 3813–3842. Crossref, Web of Science, Google Scholar
10. J. Basilico and T. Hofmann , Unifying collaborative and content-based filtering, in Proc. Twenty-First Int. Conf. Machine Learning, 2004, p. 9. Google Scholar
11. R. Bekkerman and J. Jeon , Multi-modal clustering for multimedia collections, in 2007 IEEE Conf. Computer Vision and Pattern Recognition, 2007, pp. 1–8. Google Scholar
12. R. P. Biuk-Aghai, S. Fong and Y. W. Si , Design of a recommender system for mobile tourism multimedia selection, in 2nd Int. Conf. Internet Multimedia Services Architecture and Applications, 2008, pp. 1–6. Google Scholar
13. Y. Chen, L. Wang and M. Dong , Semi-supervised document clustering with simultaneous text representation and categorization, in Proc European Conf Machine Learning and Knowledge Discovery in Databases: Part I, 2009, pp. 211–226. Google Scholar
14. J. Chen, H. Zhang, X. He, L. Nie, W. Liu and T. Chua , Attentive collaborative filtering: Multimedia recommendation with item- and component-level attention, in Proc. 40th Int. ACM SIGIR Conf. Research and Development in Information Retrieval, 2017, pp. 335–344. Google Scholar
15. D. D’Auria, D. Di Mauro, D. M. Calandra and F. Cutugno , Caruso: Interactive headphones for a dynamic 3D audio application in the cultural heritage context, in Proc. IEEE 15th Int. Conf. Information Reuse and Integration, 2014, pp. 525–528. Google Scholar
16. D. D’Auria, D. Di Mauro, D. M. Calandra and F. Cutugno , A 3D audio augmented reality system for a cultural heritage management and fruition, J. Digit. Inf. Manag. 13 (4) (2015) 203–209. Google Scholar
17. D. D’Auria, F. Persia and B. Siciliano , A low-cost haptic system for wrist rehabilitation, in IEEE Int. Conf. Information Reuse and Integration, 2015, pp. 491–495. Google Scholar
18. D. D’Auria, F. Persia and B. Siciliano , Human–computer interaction in healthcare: How to support patients during their wrist rehabilitation, in IEEE Tenth Int. Conf. Semantic Computing, 2016, pp. 325–328. Google Scholar
19. D. M. Calandra, D. Di Mauro, D. D’Auria and F. Cutugno , E.y.e. c. u.: An emotional eye tracker for cultural heritage support, in Empowering Organizations, eds. T. Torre, A. M. Braccini and R. Spinelli (Springer International Publishing, Cham, 2016), pp. 161–172. Crossref, Google Scholar
20. I. S. Dhillon, S. Mallela and D. S. Modha , Information-theoretic co-clustering, in Proc. Ninth ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2003, pp. 89–98. Google Scholar
21. B. Gao, T. Y. Liu and W. Y. Ma , Star-structured high-order heterogeneous data co-clustering based on consistent information theory, in Sixth Int. Conf. Data Mining, 2006, pp. 880–884. Google Scholar
22. M. Ge and F. Persia , A survey of multimedia recommender systems: Challenges and opportunities, Int. J. Semant. Comput. 11 (3) (2017) 411–428. Link, Web of Science, Google Scholar
23. M. Ge and F. Persia , Research challenges in multimedia recommender systems, in Proc. 11th IEEE Int. Conf. Semantic Computing, 2017, pp. 344–347. Google Scholar
24. G. Greco, A. Guzzo and L. Pontieri , Coclustering multiple heterogeneous domains: Linear combinations and agreements, IEEE Trans. Knowl. Data Eng. 22 (12) (2010) 1649–1663. Crossref, Web of Science, Google Scholar
25. F. Gedikli, F. Bagdat, M. Ge and D. Jannach , RF-REC: Fast and accurate computation of recommendations based on rating frequencies, in Proc. IEEE Conf. Commerce and Enterprise Computing, 2011, pp. 50–57. Google Scholar
26. H. Gilmore , Product Conformance Cost Quality Progress 7 (15) (1974) 16–19. Google Scholar
27. C. Gronroos, Strategic management and marketing in the service sector, Marketing Science Institute, 1983. Google Scholar
28. D. Ienco, R. G. Pensa and R. Meo , Parameter-free hierarchical co-clustering by $n$ -Ary splits, in Proc. European Conf. Machine Learning and Knowledge Discovery in Databases — Volume Part I, 2009, pp. 16–19. Google Scholar
29. K. Juszczyszyn, P. Kazienko and K. Musial , Personalized ontology-based recommender systems for multimedia objects, in Agent and Multi-Agent Technology for Internet and Enterprise Systems (Springer, Berlin, 2010), pp. 275–292. Google Scholar
30. J. Juran and A. Godfrey , Quality Control Handbook, 5th edn. (McGraw-Hill, New York, 1998). Google Scholar
31. B. Long, Z. Zhang, X. Wu and P. S. Yu , Spectral clustering for multi-type relational data, in Proc. 23rd Int. Conf. Machine Learning, 2006, pp. 585–592. Google Scholar
32. B. Long, Z. M. Zhang and P. S. Yu , A probabilistic framework for relational clustering, in Proc. 13th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2007, pp. 470–479. Google Scholar
33. M. G. Manzato and R. Goularte , A multimedia recommender system based on enriched user profiles, in Proc. 27th Annual ACM Symp. Applied Computing, 2012, pp. 975–980. Google Scholar
34. S. F. Mohsin, M. A. Jinnah and R. U. Rashid , Web-based multimedia recommendation system for e-learning website, Int. J. Adv. Netw. Appl. 1 (4) (2010) 217–233. Google Scholar
35. S. Moursi, M. Elsakhawy and H. Ghenniwa , Agent oriented media recommender system utilizing Smart multimedia, in Proc. 15th Int. Conf. Computer Supported Cooperative Work in Design, 2011, pp. 437–444. Google Scholar
36. V. Moscato, A. Picariello, F. Persia and A. Penta , A system for automatic image categorization, in IEEE Int. Conf. Semantic Computing, 2009, pp. 624–629. Google Scholar
37. W.-I. Park, S. Kang and Y.-K. Kim , A personalized multimedia contents recommendation using a psychological model, Comput. Sci. Inf. Syst. 9 (1) (2012) 1–21. Crossref, Web of Science, Google Scholar
38. M. J. Pazzani and D. Billsus , Content-based recommendation systems, in The Adaptive Web, eds. P. Brusilovsky, A. Kobsa and W. Nejdl , Lecture Notes in Computer Science, Vol. 4321 (Springer-Verlag, Berlin, Heidelberg, 2007), pp. 325–341. Crossref, Google Scholar
39. Z. U. Rehman, F. Hussain and O. Hussain , Frequency-based similarity measure for multimedia recommender systems, Multimedia Syst. 19 (2) (2013) 95–102. Crossref, Web of Science, Google Scholar
40. F. Sanchez, M. Barrilero, S. Uribe, F. Alvarez, A. Tena and J. M. Menendez , Social and content hybrid image recommender system for mobile social networks, Mobile Netw. Appl. 17 (2012) 782–795. Crossref, Web of Science, Google Scholar
41. H. Yildirim and M. S. Krishnamoorthy , A random walk method for alleviating the sparsity problem in collaborative filtering, in Proc. ACM Conf. Recommender Systems, 2008, pp. 131–138. Google Scholar