Framework and Assessment of Conversational Virtual Humans as Role-players in Simulated Social Encounters with People
DOI:
https://doi.org/10.53555/nnbma.v2i2.109Keywords:
Virtual human communications, virtual role-players, simulated social interaction, communication, conversational modelAbstract
As technology advances, people increasingly converse with computer conversational agents serving in a personal assistant role such as Siri or Cortona. Beyond the personal assistant role, a virtual humans may substitute for a person in another role during a social encounter in a simulated environment. From an experiential or pedantic training perspective, the resulting simulated social encounter may provide a forum by which real people may gain social skills. This article reviews advances in technology and science of virtual humans in faceto-face social encounters with people. An assessment framework is proposed based on a modification of the Schroeder, Adesope, & Gilbert framework [1]. Conversational attributes include verbal content and expression and nonverbal behavior such as head position or inclination, micro and macro facial or body emotional expression, eye gaze and direction, and other facial and other bodily gestures. Using that framework, ten instantiations of virtual humans are analyzed with respect to simulating a person in a bi-directional conversation. Findings and discussion address the question: can a virtual human truly be a social partner with a person for the purpose of social skills training? Within a social skills learning setting, analysis raises concern about the impact of textual data concurrently displayed with verbal message content and expression, and nonverbal behavior. Suggested future advanced technology research includes virtual human eye tracking technology in order to a better understand the cognitive science and behavioral responses associated with periods of engagement and boredom during interactions with a person.
References
Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013). How Effective are Pedagogical Agents for Learning? A Meta-Analytic Review. Journal of Educational Computing Research, 49(1), 1–39
Holsbrink-Engels, G. A. (2001). Using a computer learning environment for initial training in dealing with social-communicative problems. British Journal of Educational Technology, 32(1), 53–67
Irish, J. E. N. (2013). Can I sit here? A review of the literature supporting the use of single-user virtual environments to help adolescents with autism learn appropriate social communication skills. Computers in Human Behavior, 29(5), A17–A24
Johnson, W. L. (2014). Using Virtual Role-Play to Prepare for Cross-Cultural Communication. In T. Ahram, W. Karwowski, & T. Marek (Eds.), Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014. Krakow, Poland
Lane, H. C. (2010). Individualized Virtual Humans for Social Skills Training 2 Towards Individualized Practice with Virtual Humans. AI Magazine, 273–274
Ogan, A., Aleven, V., Kim, J., & Jones, C. (2010). Intercultural negotiation with virtual humans: The effect of social goals on gameplay and learning. In Intelligent Tutoring Systems (pp. 174–183). Springer Berlin Heidelberg
Parsons, S., & Mitchell, P. (2002). The potential of virtual reality in social skills training for people with autistic spectrum disorders. Journal of Intellectual Disability Research, 46(5), 430–43
Tartaro, A., Cassell, J., Ratz, C., Lira, J., & Nanclares-Nogués. (2014). Accessing Peer Social Interaction: Using Authorable Virtual Peer Technology as a Component of a Group Social Skills Intervention Program. ACM Transactions on Accessible Computing, 6(1), 2–2
Gratch, J., Rickel, J., Andre, E., Cassell, J., Petajan, E., & Badler, N. (2002). Creating Interactive Virtual Humans: Some Assembly Required. IEEE Intelligent Systems, (July/Auguest 2002)
Campbell, J., Hays, M. J., Core, M., Birch, M., Bosack, M., & Clark, R. E. (2011). Interpersonal and Leadership Skills: Using Virtual Humans to Teach New Officers. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011. Orlando, FL
Frank, G., Guinn, C., Hubal, R., Carolina, N., Pope, P., Stanford, M., & Lamm-weisel, D. (2002). JUSTTALK: An Application of Responsive Virtual Human Technology. In Proceedings of Interservice/Industry Training, Simulation and Education Conference. Orlando, FL
Mykoniatis, K., Angelopoulou, A., Proctor, M. D., & Karwowski, W. (2014). Virtual Humans for Interpersonal and Communication Skills ’ Training in Crime Investigations. In Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments (pp. 282–292). Switzerland: Springer International Publishing
Sotomayor, T. M., & Proctor, M. D. (2009). Assessing Combat Medic Knowledge and Transfer Effects Resulting from Alternative Training Treatments. The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, 6(3), 121–134
Talbot, T. B., Sagae, K., John, B., & Rizzo, A. (2012). Designing Useful Virtual Standardized Patient Encounters. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2012
Johnson, W. L. (2010). Serious Use of a Serious Game for Language Learning. International Journal of Artificial Intelligence in Education, 20(2), 175–195
Proctor, M., Cardona, G., & Hollander, C. (2008). Using Simulation to investigate a Non-Anthropomorphic Framework for Communications within a Human-Agent War-Fighting Team. The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, 5(2), 122–138
McFall, R. M. (1982). A review and reformation of the concept of social skills. Behavioral Assessment, 4(1), 1–33
Segrin, C., & Givertz, M. (2003). Methods of Social Skills Training and Development. In J. O. Greene & B. R. Burlesono (Eds.), Handbook of Communication and Social Interaction Skills. Mahwah, NJ: Lawrence Erlbaum Associates, Inc
Stamp, G. H. (1999). A qualitatively constructed interpersonal communication model - A grounded theory analysis. Human Communication Research, 25(4), 531–547
Weizenbaum, J. (1966). ELIZA--A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM, 9(1), 36–45
Whitworth, B. (2005). Polite computing. Behaviour & Information Technology, 24(5), 353–363
Sproull, L., Subramani, M., Kiesler, S., Walker, J. H., & Waters, K. (1996). When the Interface Is a Face. Human-Computer Interaction, 11, 97–124
Takeuchi, A., & Naito, T. (1995). Situated Facial Displays: Towards Social Interaction. In I. Katz, R. Mack, L. Marks, M. B. Rosson, & J. Nielsen (Eds.), Human Factors in Computing Systems: CHI’95 Electronic Conference Proceedings. ACM
Yee, N., Bailenson, J. N., & Rickertsen, K. (2007). A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’07 (p. 1). San Jose, CA: ACM Press
Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., & Sasse, M. A. (2003). The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment. In Proceedings of the conference on Human factors in computing systems - CHI ’03 (p. 529). New York, New York, USA: ACM Press
Beale, R., & Creed, C. (2009). Affective interaction: How emotional agents affect users. International Journal of Human-Computer Studies, 67(9), 755–776
De Melo, C. M., Carnevale, P., & Gratch, J. (2010). The Influence of Emotions in Embodied Agents on Human Decision-Making. In Intelligent Virtual Agents 2010 (pp. 357–370)
Bickmore, T., & Cassell, J. (2001). Relational Agents: A Model and Implementation of Building User Trust. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 396–403). New York
Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., Werf, R. J. Van Der, & Morency, L. (2006). Virtual Rapport. In 6th International Conference on Intelligent Virtual Agents. Marina del Rey, CA
Bailenson, J. N., Blascovich, J., Beall, A. C., & Loomis, J. M. (2003). Interpersonal Distance in Immersive Virtual Environments. Personality and Social Psychology Bulletin, 29(7), 1–15
Bickmore, T., Gruber, A., & Picard, R. (2005). Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Education and Couseling, 59(1), 21–30
Cassell, J., Stocky, T., Bickmore, T., Gao, Y., Nakano, Y., Ryokai, K., … Vilhjálmsson, H. (2002). MACK: Media lab Autonomous Conversational Kiosk. In Proceedings of IMAGINA’02. Monte Carlo
Gratch, J., Wang, N., Gerten, J., Fast, E., & Duffy, R. (2007). Creating Rapport with Virtual Agents. In International Conference on Intelligent Virtual Agents. Paris, France
Cassell, J., Bickmore, T., Billinghurst, M., Leaper, C., Chang, K., Vilhjálmsson, H., & Yan, H. (1999). Embodiment in Conversational Interfaces: Rea. In SIGCHI conference on Human Factors in Computing Systems (pp. 520–527)
Kopp, S., Gesellensetter, L., Krämer, N. C., & Wachsmuth, I. (2005). A conversational agent as museum guide - Design and evaluation of a real-world application. In Intelligent Virtual Agents 2005 (pp. 329–343)
Bickmore, T. W., Vardoulakis, L. M. P., & Schulman, D. (2013). Tinker: A relational agent museum guide. Autonomous Agents and Multi-Agent Systems, 27(2), 254–276
Swartout, W., Traum, D., Artstein, R., Noren, D., Debevec, P., Bronnenkant, K., … White, K. (2010). Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides. In Intelligent Virtual Agents 2010 (pp. 286–300)
Kim, J. M., Hill, R. W., Technologies, C., Lane, H. C., Forbell, E., Core, M., … Hart, J. (2009). BiLAT: A Game-Based Environment for Practicing Negotiation in a Cultural Context. International Journal of Artificial Intelligence in Education, 19, 289–308
Baur, T., Damian, I., Gebhard, P., Porayska-Pomsta, K., & Andre, E. (2013). A Job Interview Simulation: Social Cue-based Interaction with a Virtual Character. In 2013 ASE/IEEE International Confernence on Social Computing (Socialcom 2013)
Tartaro, A., Cassell, J., Ratz, C., Lira, J., & Nanclares-Nogués. (2014). Accessing Peer Social Interaction: Using Authorable Virtual Peer Technology as a Component of a Group Social Skills Intervention Program. ACM Transactions on Accessible Computing, 6(1), 2–29
Smith, M. J., Ginger, E. J., Wright, M., Wright, K., Boteler Humm, L., Olsen, D., … Nanclares-Nogués. (2014). Virtual Reality Job Interview Training for Individuals With Psychiatric Disabilities. The Journal of Nervous and Mental Disease, 202(5), 659–667
Argyle, M., & Cook, M. (1976). Gaze and Mutual Gaze. Oxford, England: Cambridge University Press
Argyle, M., & Dean, J. (1965). Eye-Contact, Distance and Affiliation. Sociometry, 28(3), 289–304
Hays, M. J., Campbell, J. C., Trimmer, M. A., Poore, J. C., & Webb, A. K. (2012). Can Role-Play with Virtual Humans Teach Interpersonal Skills? In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2012. Orlando, FL
Devault, D., Artstein, R., Benn, G., Dey, T., Fast, E., Gainer, A., … Morency, L. (2014). SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support. In 13th International Conference on Autonomous Agents and Multi-Agent Systems, International Foundation for Autonomous Agents and Multiagent Systems (pp. 1061–1068)
Morency, L., Stratou, G., Devault, D., Hartholt, A., Lhommet, M., Lucas, G., … Rizzo, A. (2015). SimSensei Demonstration: A Perceptive Virtual Human Interviewer for Healthcare Applications. In Twenty-Ninth AAAI Conference on Artificial Intelligence
Qu, C., Brinkman, W. P., Ling, Y., Wiggers, P., & Heynderickx, I. (2014). Conversations with a virtual human: Synthetic emotions and human responses. Computers in Human Behavior, 34, 58–68
Smith, M. J., Ginger, E. J., Wright, K., Wright, M. A., Taylor, J. L., Humm, L. B., … Fleming, M. F. (2014). Virtual reality job interview training in adults with autism spectrum disorder. Journal of Autism and Developmental Disorders, 44(10), 2450–2463
Johnson, W. L., & Valente, A. (2008). Tactical Language and Culture Training Systems: Using Artificial Intelligence to Teach Foreign Languages and Cultures. In AAAI (pp. 1632–1639)
Baur, T., Damian, I., Gebhard, P., Porayska-Pomsta, K., & Andre, E. (2013). A Job Interview Simulation: Social Cue-based Interaction with a Virtual Character. In 2013 ASE/IEEE International Confernence on Social Computing (Socialcom 2013)
Chollet, M., Morency, L., Shapiro, A., Scherer, S., & Angeles, L. (2015). Exploring Feedback Strategies to Improve Public Speaking: An Interactive Virtual Audience Framework, 1143–1154
Hoque, M., Courgeon, M., & Martin, J. (2013). Mach: My automated conversation coach. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing (pp. 697–706)
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Terms & Condition
Submission -
Author can submit the manuscript through our online submission process or email us at the designated email id in contact details.
The other mode of submission not accepted than online and email.
Before submission please read the submission guidelines.
NN Publication accepts only article submitted in pdf/doc/docx/rtf file format. Another format except given file formats will no be considered .
Author will be responsible for the error mistakes in the submission files. The minor changes can be done without any cost after publication. But for major changes NN Publication may charges you the editing charges.
Publication (Online) -
The online publication is scheduled on last date of every month, but it can be delayed by 24 to 48 hours due to editorial process if huge number of articles comes to publish in single issue.
Automatic notification email will be sent to the all users on publication of an issue, so its author’s duty to check their email inbox or SPAM folder to get this notification.
After publication of article author can not withdraw their article.
If editor’s found any issue after publication of article then the NN Publication have the authority to remove the article from online website.
No refund will be provided after online publication of article.
Publication (Print) -
The print copy publication are sent as per the author’s request after 2 weeks of online publication of that issue.
NN Publication will ship the article by India Post and provide the consignment number on dispatch of print copy.
NN Publication follows all the guidelines of delivery provided by India Post and hence not responsible for delay in delivery due to any kind of reasons.
Refund of hard copy will not be provided after dispatch or print of the journal.
NN Publication will be responsible for raise a complain if there is any issue occurs in delivery, but still will not be responsible for providing the refund.
NN Publication will be responsible to resend the print copy only and only if the print copy is lost or print copy is damaged in delivery / or there is delay more than 6 months.
According to India Post the delivery should be completed with in 1-3 weeks after dispatch of articles.
Privacy Policy-
NN Publication uses the email ids of authors and editors and readers for sending editorial or publication notification only, we do not reveal or sell the email ids to any other website or company.
