This research aims to develop a system that examines and reacts to the changing behaviors and emotions of individuals in order to improve their shopping experience. The system is able to track emotions in real time at different touchpoints in a store and control a set of networked devices to configure the sensing space and all provided services responsive to the customers’ needs. This paper describes the general approach adopted to design the overall system and illustrates in detail the module prototyped to understand the users’ emotions through the analysis of facial expressions.

1. [1] L. L. Berry, L. P. Carbone, and S. H. Haeckel, “Managing the total customer experience,” MIT Sloan Management Review, Vol.43, No.3, pp. 85-90, 2002.
2. [2] C. Ofir and I. Simonson, “The effect of stating expectations on customer satisfaction and shopping experience,” J. of Marketing Research, Vol.44, No.1, pp. 164-174, 2007.
3. [3] D. W. Wallace, J. L. Giese, and J. L. Johnson, “Customer retailer loyalty in the context of multiple channel strategies,” J. of Retailing, Vol.80, No.4, pp. 249-263, 2004.
4. [4] J. B. Pine and J. B. Gilmore, “The Experience Economy,” Harvard Business School Press, 1999.
5. [5] H. J. Chen-Yu and D. H. Kincade, “Effects of product image at three stages of the consumer decision process for apparel products: Alternative evaluation, purchase and post-purchase,” J. of Fashion Marketing and Management: An Int. Journal, Vol.5, No.1, pp. 29-43, 2001.
6. [6] D. Grewal, M. Levy, and V. Kumar, “Customer experience management in retailing: An organizing framework,” J. of Retailing, Vol.85, No.1, pp. 1-14, 2009.
7. [7] L. G. Zomerdijk and C. A. Voss, “Service design for experience centric services,” J. of Service Research, Vol.13, No.1, pp. 67-82, 2010.
8. [8] L. Giraldi, M. Mengoni, and M. Bevilacqua, “How to Enhance Customer Experience in Retail: Investigations Through a Case Study,” Transdisciplinary Engineering: Crossing Boundaries: Proc. of the 23rd ISPE Inc. Int. Conf. on Transdisciplinary Engineering, Vol.4, pp. 381-390, 2016.
9. [9] S. Ceccacci, L. Giraldi, and M. Mengoni, “From customer experience to product design: Reasons to introduce a holistic design approach,” Proc. of the Int. Conf. on Engineering Design, ICED, Vol.4 (DS87-4), pp. 463-472, 2017.
10. [10] K. A. Machleit and S. A. Eroglu, “Describing and Measuring Emotional Response to Shopping Experience,” J. of Business Research, Vol.49, No.2, pp. 101-111, 2000.
11. [11] C. Meyer and A. Schwager, “Understanding Customer Experience,” Harvard Business Review, Vol.85, No.2, pp. 117-126, 2007.
12. [12] M. N. Khuong and V. N. B. Tram, “The Effects of Emotional Marketing on Consumer Product Perception, Brand Awareness and Purchase Decision – A Study in Ho Chi Minh City, Vietnam,” J. of Economics, Business and Management, Vol.3, No.5, pp. 524-530, 2015.
13. [13] C. Achar, J. So, N. Agrawal, and A. Duhachek, “What we feel and why we buy: the influence of emotions on consumer decision-making,” Current Opinion in Psychology, Vol.10, pp. 166-170, 2016.
14. [14] P. Frow and A. Payne, “Towards the ‘perfect’ customer experience,” J. of Brand Management, Vol.15, No.2, pp. 89-101, 2007.
15. [15] P. C. Verhoef, K. N. Lemon, A. Parasuraman, A. Roggeveen, M. Tsiros, and L. A. Schlesinger, “Customer experience creation: Determinants, dynamics and management strategies,” J. of retailing, Vol.85, No.1, pp. 31-41, 2009.
16. [16] J. Teixeira, L. Patrício, N. J. Nunes, L. Nóbrega, R. P. Fisk, and L. Constantine, “Customer experience modeling: from customer experience to service design,” J. of Service Management, Vol.23, No.3, pp. 362-376, 2012.
17. [17] B. Edvardsson, P. Kristensson, P. Magnusson, and E. Sundström, “Customer integration within service development – A review of methods and an analysis of insitu and exsitu contributions,” Technovation, Vol.32, No.7, pp. 419-429, 2012.
18. [18] M. Maguire, “Methods to support human-centred design,” Int. J. of Human-Computer Studies, Vol.55, No.4, pp. 587-634, 2001.
19. [19] J. Füller and K. Matzler, “Virtual product experience and customer participation – A chance for customer-centred, really new products,” Technovation, Vol.27, No.6, pp. 378-387, 2007.
20. [20] A. Stein and B. Ramaseshan, “Towards the identification of customer experience touch point elements,” J. of Retailing and Consumer Services, Vol.30, pp. 8-19, 2016.
21. [21] B. Stauss and B. Weinlich, “Process-oriented measurement of service quality: Applying the sequential incident technique,” European J. of Marketing, Vol.31, No.1, pp. 33-55, 1997.
22. [22] G. Das and G. Varshneya, “Consumer emotions: Determinants and outcomes in a shopping mall,” J. of Retailing and Consumer Services, Vol.38, pp. 177-185, 2017.
23. [23] B. J. Babin, M. Griffin, A. Borges, and J. S. Boles, “Negative emotions, value and relationships: Differences between women and men,” J. of Retailing and Consumer Services, Vol.20, No.5, pp. 471-478, 2013.
24. [24] J. N. Bailenson, E. D. Pontikakis, I. B. Mauss, J. J. Gross, M. E. Jabon, A. C. Cendri, et al., “Real-time classification of evoked emotions using facial feature tracking and physiological responses,” Int. J. of Human-Computer Studies, Vol.66, No.5, pp. 303-317, 2008.
25. [25] D. Borth, R. Ji, T. Chen, T. Breuel, and S. F. Chang, “Large-scale visual sentiment ontology and detectors using adjective noun pairs,” Proc. of the 21st ACM Int. Conf. on Multimedia, pp. 223-232, 2013.
26. [26] Affdex, https://www.affectiva.com/product/affdex-for-market-research/ [Retrieved April 5, 2018]
27. [27] P. Ekman and D. Keltner, “Universal facial expressions of emotion,” California Mental Health Research Digest, Vol.8, No.4, pp. 151-158, 1970.
28. [28] P. Ekman and W. V. Wallace, “Manual for the Facial Action Coding System,” Palo Alto: Consulting Psychologists Press, 1977.
29. [29] Google APIs, Speech API – Speech Recognition, Google Cloud Platform (n.d.), https://cloud.google.com/speech/ [Retrieved February 9, 2017]
30. [30] IBM Watson Developer Cloud, https://www.ibm.com/watson/developercloud/doc/tone-analyzer/science.html [Retrieved February 9, 2017]
31. [31] D. Ververidis and C. Kotropoulos, “Emotional speech recognition: resources, features, and methods,” Speech Commun., Vol.48, No.9, pp. 1162-1181, 2006.
32. [32] PeakProfiling, https://www.peakprofiling.com/technology.html [Retrieved February 20, 2018]
33. [33] D. S. Quintana, A. J. Guastella, T. Outhred, I. B. Hickie, and A. H. Kemp, “Heart rate variability is associated with emotion recognition: direct evidence for a relationship between the autonomic nervous system and social cognition,” Int. J. of Psychophysiology, Vol.86, No.2, pp. 168-172, 2012.
34. [34] M. Hepp, “Goodrelations: An ontology for describing products and services offers on the web,” Int. Conf. on Knowledge Engineering and Knowledge Management, pp. 329-346, 2008.
35. [35] WHO, “International classification of functioning, disability and health,” World Health Organization (WHO), Geneva, Switzerland, 2001.
36. [36] M. A. Max-Neef, “Human scale development: conception, application and further reflections,” Apex Press, 1991.
37. [37] A. S. Ullah, M. Sato, M. Watanabe, and M. M. Rashid, “Integrating CAD, TRIZ, and Customer Needs,” Int. J. Automation Technol., Vol.10, No.2, 2016.
38. [38] D. Hay and K. A. Helay, “Defining Business Rules – What Are They Really?,” GUIDE Business Rules Project Report (1995), 3rd edition, 2000.
39. [39] F. Gailly and G. L. Geerts, “Ontology-Driven Business Rule Specification,” J. of Information Systems, Vol.27, No.1, pp. 79-104, 2013.
40. [40] T. Eiter, G. Ianni, A. Polleres, R. Schindlauer, and H. Tompits, “Reasoning with Rules and Ontologies,” Reasoning web, pp. 93-127, 2006.
41. [41] AudioProfiling Engine – Technology, http://www.audioprofiling.com/technology.html?lang=en [Retrieved February 9, 2017]
42. [42] SensauraTech API, http://api.sensauratech.com/ [Retrieved February 9, 2017]