World Scientific
  • Search
Skip main navigation

Cookies Notification

We use cookies on this site to enhance your user experience. By continuing to browse the site, you consent to the use of our cookies. Learn More
Our website is made possible by displaying certain online content using javascript.
In order to view the full content, please disable your ad blocker or whitelist our website

System Upgrade on Tue, Oct 25th, 2022 at 2am (EDT)

Existing users will be able to log into the site and access content. However, E-commerce and registration of new users may not be available for up to 12 hours.
For online purchase, please visit us again. Contact us at [email protected] for any enquiries.

Evaluation Techniques for Chatbot Usability: A Systematic Mapping Study by:21 (Source: Crossref)
    This article is part of the issue:

    Background: The use of chatbots has increased considerably in recent years. These are used in different areas and by a wide variety of users. Due to this fact, it is essential to incorporate usability in their development. Aim: Our objective is to identify the state-of-the-art in chatbot usability and applied human–computer interaction techniques, to analyze how to evaluate chatbot usability. Method: We have conducted a systematic mapping study, by searching the main scientific databases. The search retrieved 170 references and 21 articles were retained as primary studies. Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots. Conclusions: Chatbot usability is still a very incipient field of research where the published studies are mainly surveys, usability tests, and rather informal experimental studies. Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.


    • 1. M. Jain, R. Kota, P. Kumar and S. N. Patel , Convey: Exploring the use of a context view for chatbots, in Proc. 2018 CHI Conf. Human Factors in Computing Systems, 2018, p. 468. CrossrefGoogle Scholar
    • 2. ISO 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs)–Part II Guidance on Usability, 1998. Google Scholar
    • 3. Q. N. Nguyen and A. Sidorova , Understanding user interactions with a chatbot: A self-determination theory approach, in Americas Conf. Information Systems 2018: Digital Disruption, New Orleans, 2018, pp. 1–5. Google Scholar
    • 4. K. Panetta, Gartner’s top 10 strategic technology trends for 2017, Google Scholar
    • 5. J. Pereira and O. Diaz , A quality analysis of Facebook messenger’s most popular Chatbots, in Proc. 33rd Annual ACM Symp. Applied Computing, 2018, pp. 2144–2150. CrossrefGoogle Scholar
    • 6. C. Messina, 2016 will be the year of conversational commerce, Google Scholar
    • 7. J. Lester, K. Branting and B. Mott , Conversational Agents (The Practical Handbook of Internet Computing, Chapman and Hall/CRC, 2004), pp. 220–240. Google Scholar
    • 8. LOLA Asistente Virtual de la Universidad de Murcia para Ayudar en el Proceso de Preinscripción y Matrícula, 2018, Google Scholar
    • 9. S. Pérez-Soler, E. Guerra, J. de Lara and F. Jurado , The rise of the (modelling) bots: Towards assisted modelling via social networks, in Proc. 32nd IEEE/ACM Int.Conf. Automated Software Engineering, 2017, pp. 723–728. CrossrefGoogle Scholar
    • 10. L. Sullivan, Facebook Chatbots Hit 70% Failure Rate as Consumers Warm Up to the Tech, 2017, Google Scholar
    • 11. L. C. Klopfenstein, S. Delpriori, S. Malatini and A. Bogliolo , The rise of bots: A survey of conversational interfaces, patterns, and paradigms, in Proc. Conf. Designing Interactive Systems, 2017, pp. 555–565. CrossrefGoogle Scholar
    • 12. S. Pérez-Soler, E. Guerra and J. de Lara , Collaborative modeling and group decision making using chatbots in social networks, IEEE Softw. 35(6) (2018) 48–54. Crossref, ISIGoogle Scholar
    • 13. R. Ren, J. W. Castro, S. T. Acuña and J. de Lara , Usability of chatbots: A systematic mapping study, in Proc. 31st Int. Conf. Software Engineering and Knowledge Engineering, 2019, pp. 479–484. CrossrefGoogle Scholar
    • 14. K. Ramesh, S. Ravishankaran, A. Joshi and K. Chandrasekaran , A survey of design techniques for conversational agents, in eds. S. KaushikD. GuptaL. KharbD. Chahal, Information, Communication and Computing Technology, Communications in Computer and Information Science, Vol. 750, Springer, 2017, pp. 336–350. CrossrefGoogle Scholar
    • 15. L. Laranjo, A. G. Dunn, H. L. Tong, A. B. Kocaballi, J. Chen, R. Bashir and D. Surian, B. Gallego, F. Magrabi, A. Y. Lau and E. Coiera , Conversational agents in healthcare: A systematic review, J. Amer. Med. Inf. Assoc. 25(9) (2018) 1248–1258. Crossref, ISIGoogle Scholar
    • 16. K. Petersen, R. Feldt, S. Mujtaba and M. Mattsson , Systematic mapping studies in software engineering, in Proc. 12th Int. Conf. Evaluation and Assessment in Software Engineering, Italy, 2008, pp. 68–77. CrossrefGoogle Scholar
    • 17. J. W. Castro and S. T. Acuña , Comparativa de selección de estudios primarios en una revisión sistemática, in Proc. 16th Jornadas de Ingeniería del Software y Bases de Datos, 2011, pp. 319–332. Google Scholar
    • 18. I. Lopatovska, K. Rink, I. Knight, K. Raines, K. Cosenza, H. Williams, P. Sorsche, D. Hirsch, Q. Li and A. Martinez , Talk to me: Exploring user interactions with the Amazon Alexa, J. Librariansh. Inf. Sci. 51(4) (2019) 984–997. Crossref, ISIGoogle Scholar
    • 19. A. Cheng, V. Raghavaraju, J. Kanugo, Y. P. Handrianto and Y. Shang , Development and evaluation of a healthy coping voice interface application using the Google home for elderly patients with type 2 diabetes, in Proc. 15th IEEE Annual Consumer Communications & Networking Conf., 2018, pp. 1–5. CrossrefGoogle Scholar
    • 20. M.-L. Chen and H.-C. Wang , How personal experience and technical knowledge affect using conversational agents, in Proc. 23rd Int. Conf. Intelligent User Interfaces Companion, 2018, Article No. 53. CrossrefGoogle Scholar
    • 21. C. Sinoo, S. van der Pal, O. A. Blanson, A. Keizer, B. P. B. Bierman, R. Looije and M. A. Neerincx , Friendship with a robot: Children’s perception of similarity between a robot’s physical and virtual embodiment that supports diabetes self-management, Patient Educ. Counseling 101(7) (2018) 1248–1255. Crossref, ISIGoogle Scholar
    • 22. J. Pérez, Y. Sánchez, F. J. Serón and E. Cerezo , Interacting with a semantic affective ECA, in Intelligent Virtual Agents, LNCS Vol. 10498, 2017, pp. 374–384. CrossrefGoogle Scholar
    • 23. A. Alghamdi, M. Owda and K. Crockett , Natural language interface to relational database (NLI-RDB) through object relational mapping (ORM), in Advances in Intelligent Systems and Computing, Vol. 513, 2017. CrossrefGoogle Scholar
    • 24. M. L. Tielman, M. A. Neerincx, R. Bidarra, B. Kybartas and W. P. Brinkman , A therapy system for post-traumatic stress disorder using a virtual agent and virtual storytelling to reconstruct traumatic memories, J. Med. Syst. 41(8) (2017) 125. Crossref, ISIGoogle Scholar
    • 25. A. Preece, W. Webberley, D. Braines, E. G. Zaroukian and J. Z. Bakdash , Sherlock: Experimental evaluation of a conversational agent for mobile information tasks, IEEE Trans. Human-Mach. Syst. 47(6) (2017) 1017–1028. Crossref, ISIGoogle Scholar
    • 26. N. C. Chi, O. Sparks, S. Y. Lin, A. Lazar, H. J. Thompson and G. Demiris , Pilot testing a digital pet avatar for older adults, Geriatric Nursing 38(6) (2017) 542–547. Crossref, ISIGoogle Scholar
    • 27. J. Saenz, W. Burgess, E. Gustitis, A. Mena and F. Sasangohar , The usability analysis of chatbot technologies for internal personnel communications, in Proc. 67th Annual Conf. and Expo of the Institute of Industrial and Systems Engineers, 2017, pp. 1357–1362. Google Scholar
    • 28. C. Tsiourti, J. Quintas, M. Ben-Moussa, S. Hanke, N. A. Nijdam and D. Konstantas , The CaMeLi framework — A multimodal virtual companion for older adults, in Intelligent Systems and Applications, Studies in Computational Intelligence, Vol. 751, Springer, 2018, pp. 196–217. CrossrefGoogle Scholar
    • 29. D. Elmasri and A. Maeder , A conversational agent for an online mental health intervention, in Brain and Health Informatics, LNCS Vol. 9919, 2016, pp. 243–251. CrossrefGoogle Scholar
    • 30. A. I. Niculescu, K. H. Yeo, L. F. D’Haro, S. Kim, R. Jiang and R. E. Banchs , Design and evaluation of a conversational agent for the touristic domain, in Proc. Signal and Information Processing Association Annual Summit and Conf., 2014, pp. 1–10. CrossrefGoogle Scholar
    • 31. S. Tegos, S. Demetriadis and T. Tsiatsos , A configurable conversational agent to trigger students’ productive dialogue: A pilot study in the CALL domain, Int. J. Artif. Intell. Educ. 24(1) (2014) 62–91. CrossrefGoogle Scholar
    • 32. J. A. Micoulaud-Franchi, P. Sagaspe, E. De Sevin, S. Bioulac, A. Sauteraud and P. Philip , Acceptability of embodied conversational agent in a health care context, in Intelligent Virtual Agents, LNCS Vol. 10011, 2016, pp. 416–419. CrossrefGoogle Scholar
    • 33. R. Yaghoubzadeh, K. Pitsch and S. Kopp , Adaptive grounding and dialogue management for autonomous conversational assistants for elderly users, in Intelligent Virtual Agents, LNCS Vol. 9238, 2015, pp. 28–38. CrossrefGoogle Scholar
    • 34. D. Novick and L. M. Rodríguez , Extending empirical analysis of usability and playability to multimodal computer games, in Design, User Experience, and Usability: Design Thinking and Methods, LNCS Vol. 9746, 2016, pp. 469–478. CrossrefGoogle Scholar
    • 35. K. Hornbæk , Current practice in measuring usability: Challenges to usability studies and research, Int. J. Human-Comput. Stud. 64(2) (2006) 79–102. Crossref, ISIGoogle Scholar
    • 36. ISO/IEC 25010, Systems and software engineering–systems and software quality requirements and evaluation (SQuaRE)–system and software quality models, 2010. Google Scholar
    Remember to check out the Most Cited Articles!

    Check out our titles in C++ Programming!