This demonstrator illustrates the deep learning capabilities implemented in EoT. The EoT board performs facial emotion recognition so that the doll can assess the child’s emotional display using deep learning and react accordingly with audio feedback.

All computations are local to the EoT device which reduces power consumption and tackles privacy issues.

To develop the EoT device application of the Empathic Doll demonstrator, the SDCardIO, TimeFunction, Camera, tiny_dnn, Audio, Libccv/OpenCV and LEDs EoT libraries have been used.

nVISO’s emotion engine was trained to classify the basic 5 emotions plus the neutral expression when no emotion is detected. The nViso emotion engine uses a specially designed convolutional neural network for emotion detection. In order to produce a model small  enough to run in the EoT device, the network is trained on a relatively smaller dataset comprising of 6258 images obtained from 700 different subjects.

The Smart Doll with Emotion recognition demonstrator has a configuration application. The configuration application is based on the basic software installation of the EoT board. This configuration includes in the EoT side the Bootloader application and the Pulga MQTT broker, whereas on the user side it will be necessary to run the MQTT Client application developed for PC or Android. Both applications can be used to configure the Wi-Fi chip for the configuration mode, to upload and flash the demonstrator and to download the text file which contains the emotions detected during the session.

---

Smart Doll – Concept http://eyesofthings.eu/?page_id=1888
Smart Doll – Results http://eyesofthings.eu/?page_id=1886