The application software for the museum audio tour demonstrator consists of two individual applications. The first one is running on the EoT device and the second one on a mobile companion device, like a smartphone or tablet, supported by a web CMS.
This demonstrator uses the following EoT libraries, developed in the workpackage_3: Audio, LEDs, MQTT Client, Wi-Fi functions, JSON parser, SDCardIO, HistogramMatching and Quirc.
These libraries have been used to build four modules: camera and ISP, painting recognition, network communication and application logic.
The painting recognition module is called for each incoming camera frame when the device is in “painting recognition” mode. This module take a camera image as input and provide a painting ID as an output. If no painting has been identified in the current frame, a specific ID “No painting” is returned.
To this aim, the following functionalities have been implemented:
- Keypoint extraction: This functionality takes the camera image as input and extract FAST keypoints from the image. Each keypoint is identified through its position (x, y) in the image.
- Keypoint classification: The classification is based on a Random Forest consisting of a number of trees. This functionality takes a keypoint position and an image as input. Each single tree takes the keypoint position as input. A tree node consist in a binary brightness test between two specific pixels in the direct neighborhood of the keypoint. According to the result of this test, the point is passed to the next node in the tree. The tree leaves consist in a probability distribution of possible keypoints as learnt in the learning phase. Thus, each keypoint will generate a probability distribution among the possible reference keypoints for each tree. The distributions of all the trees are averaged in order to provide the global probability distribution, from which the maximum is taken. If this maximum is above a threshold, the keypoint is classified as a known keypoint. Repeating this process for all the keypoints in the current frame allows for counting the number of keypoints in the current frame that correspond to a given painting.
- Geometric refinement: In order to avoid false positive, the found keypoints are tested for a correct geometric alignment. To this aim, the function attempts to recover a homography between the seen image and the reference painting, using a robust method based on the RANSAC paradigm.
The software to train the painting recognition consists of two separate applications. The first one analyses an image of a painting and extracts suitable keypoint locations. The suitability of a keypoint is defined by how reliably it can be detected in different perspectives. The program determines a stochastic approximation to that value by rendering the image in 1,000 random perspectives and counting in how many of those a given keypoint can be detected. The keypoints which have been found and scored this way are ordered by their quality and stored in a file.
The second application reads that keypoint file as well as the image and trains a random forest to recognize the keypoints of that painting. Training is also performed in a stochastic manner by again rendering the painting in random perspectives. For a given keypoint a sample of the keypoints appearance is drawn from each random view and fed to the training process of the random forest. The resulting forest is then serialized into a file, which can be read by the EoT museum guide software running on the EoT board.
The EoT Museum App is implemented as a mobile web-application using the standard web technologies, like JavaScript, HTML5 and CSS3. The app can be accessed using any mobile browser, which makes it highly accessible and allows for easy distribution, including a regular desktop browser.
Persisting data (e.g. for storing favorite exhibits) is stored in the user’s browser using localStorage.
Audio playback is implemented using HTML5 Audio elements that can be controlled by the user via UI elements for play/pause/seeking.
The Museum App is ready to be packaged as a native mobile app using Apache Cordova. This enables additional distribution through Apple App Store as well as Google Play Store.
In addition to these functionalities, the EoT museum app supports two configuration features:
- EoT-Camera Snapshot: Requesting and displaying a snapshot of the camera image. This will control the EoT-Camera and allow a Camera-preview inside the mobile app.
- Pairing of devices: Pairing of an EoT headset device and a mobile device. This is done with the camera and a dynamic QR-display.
Finally, Fluxguide developed a complete EoT Museum Guide content management system which allows the operator of the system to add, edit and delete content of the EoT visitor App. This component completes the system with a web-based, easy-to-use front end for the EoT Museum Guiding services.
During using the EoT device and the mobile app, the system will collect user data for further analysis. This data is non-personal data and collected anonymously. The users are informed about this data collection in the Terms of Service of the application. Inside the CMS there is a statistics tool for visualization of the collected data. Following information can be analysed:
- Full tracking of user activities.
- Daily, weekly, monthly statistics.
- Ranking list of most accessed exhibits.
Museum Guide – Concept http://eyesofthings.eu/?page_id=1879
Museum Guide – Results http://eyesofthings.eu/?page_id=1881