A Research Space for
Earable Computing

ABOUT

Sensory earables are increasingly becoming a mainstream compute platform with a promise to fundamentally transform personal-scale human sensing applications. Over the past few years, a number of research efforts in the ubiquitous computing domain have sought to achieve useful, engaging, and sometimes ambitious behavioural analytics with sensory earables including studies of the human face; of emotion and stress; continuous monitoring of cardiovascular function; oxygen consumption and blood flow; and tracking eating episodes as well as dietary and swallowing activities. At the same time, we have started seeing commercial efforts such as Bragi's The Dash, Bose SoundSport, Jabra Elite Sport, and Sony Xperia offering music experience augmented with sensory services including fitness tracking, real-time translations and conversational agents. Naturally, earables are becoming an intense interdisciplinary area of study with many and diverse applications including HCI, empathic communication, behavioural science, health and wellbeing, entertainment, education, and security.

This site, a voluntarily research service to the community, is intended to catalyse advancements in sensory earable technology by building an academic platform to bring together researchers, practitioners, and design experts from academia and industry to discuss, share, and shape this exciting new area of earable computing.

As a launchpad, we have developed an Open Earable Platform, eSense, and shared with 60+ academic institutions around the globe to accelerate the research in this space. This site is the nucleus of this effort and will offer details of different projects and associated papers, and datasets as they are developed by this community for the community.

The site is also the entry point of EarComp workshop, our annual meeting to discuss the progress of this place, and present a clear sense of direction for the research community to proceed in this space.

eSense OVERVIEW

eSense is a multi-sensory earable platform for personal-scale behavioural analytics research. It is a True Wireless Stereo (TWS) earbud augmented with a 6-axis inertial motion unit, a microphone, and dual mode Bluetooth (Bluetooth Classic and Bluetooth Low Energy). eSense is built with a custom-designed 15 × 15 × 3 mm PCB and composed of a Qualcomm CSR8670, a dual-mode Bluetooth audio system-on-chip (SoC) with a microphone per earbud; a InvenSense MPU6500 six-axis inertial measurement unit (IMU) including a three-axis accelerometer, a three-axis gyroscope, and a two-state button; a circular LED; associated power regulation; and battery-charging circuitry. There is no internal storage or real-time clock. It is powered by an ultra-thin 40-mAh LiPo battery. The carrier casing is equipped with a battery enabling recharging of eSense earbuds on the go (up to 3 full charges). Each earbud weights 20 g and is 18 × 20 × 20 mm.

Please check the IEEE Pervasive Computing article on eSense for more details.

If you use eSense in your research project, we would appreciate if you kindly cite the following two papers.

[1] Fahim Kawsar, Chulhong Min, Akhil Mathur, and Alessandro Montanari,. "Earables for Personal-scale Behaviour Analytics", IEEE Pervasive Computing, Volume: 17, Issue: 3, 2018
[2] Chulhong Min, Akhil Mathur and Fahim Kawsar . "Exploring Audio and Kinetic Sensing on Earable Devices", In WearSys 2018, The 16th ACM Conference on Mobile Systems, Applications, and Services (MobiSys 2018), June 2018 , Munich, Germany.

eSense RESOURCES

  • User Manual - Learn how to quickly get started with eSense.
    User Manual [PDF]
     
  • BLE Specification - Detailed structure of BLE transport and data commands
    Doc [PDF]
     
  • Data Visualiser - A node app to visualise and configure eSense sensory streams at real-time (with manual)
    Source Code | Documentation [PDF]
     
  • Android Library -A light-weight android library to build mobile apps with eSense (with manual)
    Source Code | Documentation [PDF]
     
  • eSense COMMUNITY RESOURCES

  • Flutter Plugin - The eSense Flutter plugin for both Android and iOS.
    Installation Doc - credit: CACHET Team, DTU
     
  • iOS Library - iOS library for eSense
    Source Code
    Credit: Yuuki Nishiyama, University of Tokyo
     
  • eSense iOS app - eSense client for iOS
    Download
    Credit: Yuuki Nishiyama, University of Tokyo
     
  • eSense Android app - eSense client for Android
    Source Code
    Credit: Md. Shafiqul Islam, Kyushu Institute of Technology
     
  • Publications, Projects, Models and Dataset

    60+ academic research groups from all over the globe are contributing to this research effort. The following is a living list of research papers that have used eSense or cited eSense.


    Nokia Bell Lab's Publications


    1. Earables for Personal-scale Behaviour Analytics
      Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Nokia Bell Labs
      IEEE Pervasive 2018
       
    2. Exploring Audio and Kinetic Sensing on Earable Devices
      Chulhong Min, Akhil Mathur, Fahim Kawsar, Nokia Bell Labs
      ACM WearSys 2018
       
    3. eSense: Open Earable Platform for Human Sensing
      Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Utku Gunay Acer, Marc Van den Broeck, Nokia Bell Labs
      ACM MobiSys 2018, ACM UbiComp 2018, ACM SenSys 2018 [Demo]
       
    4. Cross-modal approach for conversational well-being monitoring with multi-sensory earables
      Chulhong Min, Alessandro Montanari, Akhil Mathur, Seungchul Lee and Fahim Kawsar,Nokia Bell Labs
      ACM WellComp 2018
       
    5. Audio-Kinetic Model for Automatic Dietary Monitoring with Earable Devices
      Chulhong Min, Akhil Mathur, Fahim Kawsar, Nokia Bell Labs
      ACM MobiSys 2018 [Poster]
       
    6. Automatic Smile and Frown Recognition with Kinetic Earables
      Seungchul Lee, Chulhong Min, Akhil Mathur, Alessandro Montanari, Youngjae Chang, Junehwa Song, Fahim Kawsar, Nokia Bell Labs and KAIST
      Augmented Human 2019
       
    7. Situation-Aware Conversational Agent with Kinetic Earables
      Shin Katayama, Akhil Mathur, Tadashi Okoshi, Jin Nakazawa, Fahim Kawsar,Keio University and Nokia Bell Labs
      ACM MobiSys 2019 [Demo]
       
    8. Situation-Aware Emotion Regulation of Conversational Agents with Kinetic Earables
      Shin Katayama, Akhil Mathur, Marc Van den Broeck Tadashi Okoshi, Jin Nakazawa, Fahim Kawsar Keio University and Nokia Bell Labs
      ACM ACII 2019
       
    9. A closer look at quality-aware runtime assessment of sensing models in multi-device environments
      Chulhong Min, Alessandro Montanari, Akhil Mathur, Fahim Kawsar, Nokia Bell Labs
      ACM SenSys 2019
       
    10. An early characterisation of wearing variability on motion signals for wearables
      Chulhong Min, Akhil Mathur, Alessandro Montanari, Fahim Kawsar, Nokia Bell Labs
      ACM ISWC 2019
       
    11. The city as a personal assistant
      Utku Günay Acer, Marc Van den Broeck, Fahim Kawsar, Nokia Bell Labs
      ACM UPA 2019
       
    12. A Systematic Study of Unsupervised Domain Adaptation for Robust Human-Activity Recognition
      Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, Fahim Kawsar Nokia Bell Labs and KAIST
      ACM IMWUT 2020
       


    Research Community's Publications


    Besides Bell Labs' work, we sincerely acknowledge research groups across the world using our platform to advance state of the art. We are proud to list the following published papers using eSense or cited eSense.


    1. A comparison between audio and IMU data to detect chewing events based on an earable device
      Roya Lotfi, George Tzanetakis, Rasit Eskicioglu and Pourang Irani,
      University of Manitoba, University of Victoria
       
    2. The CAMS eSense Framework: Enabling Earable Computing for mHealth Apps and Digital Phenotyping
      Jakob E. Bardram,
      Technical University of Denmark
       
    3. ExerSense: Real-Tme Physical Exercise Segmentation, Classification, and Counting Algorithm Using an IMU Sensor
      Shun Ishii, Kizito Nkurikiyeyezu, Anna Yokokubo, Guillaume Lopez,
      Aoyama Gakuin University
       
    4. Human activity recognition using earable device
      Tahera Hossain, Md Shafiqul Islam, Md Atiqur Rahman Ahad, Sozo Inoue,
      Kyushu Institute of Technology, University of Dhaka
       
    5. Devising and evaluating wearable technology for social dynamics monitoring
      Alessandro Montanari,
      University of Cambridge
       
    6. As you are, so shall you move your head: a system-level analysis between head movements and corresponding traits and emotions
      Sharmin Akther Purabi, Rayhan Rashed, Mirajul Islam, Nahiyan Uddin, Mahmuda Naznin, and A. B. M. Alim Al Islam,
      BUET
       
    7. Using the eSense Wearable Earbud as a Light-Weight Robot Arm Controller
      Henry Odoemelem, Alexander Hölzemann, Kristof Van Laerhoven,
      University of Siegen
       
    8. EarEcho: Using Ear Canal Echo for Wearable Authentication
      Yang Gao, Wei Wang, Vir V. Phoha, Wei Sun, Zhanpeng Jin,
      University of Buffalo and Syracuse University
       
    9. Acoustic Transparency and the Changing Soundscape of Auditory Mixed Reality
      Mark McGill, Stephen Brewster, David McGookin, Graham Wilson,
      University of Glasgow
       
    10. STEAR: Robust Step Counting from Earables
      Jay Prakash, Zhijian Yang, Yu-Lin Wei, Romit Roy Choudhury,
      University of Illinois at Urbana Champaign (UIUC) and Singapore University of Technology and Design (SUTD)
       
    11. Esports Athletes and Players: A Comparative Study
      Nikita Khromov, Alexander Korotin, Andrey Lange, Anton Stepanov, Evgeny Burnaev, Andrey Somov,
      Skolkovo Institute of Science and Technology
       
    12. Towards Respiration Rate Monitoring Using an In-Ear Headphone Inertial Measurement Unit
      Tobias Röddiger, Daniel Wolffram, David Laubenstein, Matthias Budde and Michael Beigl
      Karlsruhe Institute of Technology
       
    13. Head Motion Tracking Through in-Ear Wearables
      Andrea Ferlini, Alessandro Montanari, Cecilia Mascolo and Robert Harle
      University of Cambridge, and Nokia Bell Labs
       
    14. Can Earables Support Effective User Engagement during Weight-Based Gym Exercises?
      Meeralakshmi Radhakrishnan and Archan Misra
      Singapore Management Univeristy
       
    15. EStep: Earabales as opportunity for physio-analytics
      Jay Prakash, Zhijian Yang, Yu-Lin Wei and Romit Roy Choudhury
      University of Illinois at Urbana Champaign (UIUC)
       
    16. Towards In-Ear Inertial Jaw Clenching Detection
      Siddharth Rupavatharam and Marco Gruteser
      Rutgers University
       
    17. Veers: a Case Study of Acoustical Manipulation in Walking without Sight both on Subtle and Overt Conditions
      Kohei Matsumura and Kazushi Okada
      Ritsumeikan University
       
    18. Using an in-ear device to annotate activity data across multiple wearable sensors
      Alexander Hölzemann, Henry Odoemelemand and Kristof Van Laerhoven
      University of Siegen
       
    19. Using the eSense Wearable Earbud as a Light-Weight Robot Arm Controller
      Henry Odoemelem, Alexander Hölzemann and Kristof Van Laerhoven
      University of Siegen
       
    20. A data sharing platform for earables research
      Jovan Powar and Alastair R Beresford
      University of Cambridge
       
    21. Enhanced gesture sensing using battery-less wearable motion trackers
      Huy Vu Tran,
      Singapore Management University
       
    22. Augmenting TV Viewing using Acoustically Transparent Auditory Headsets
      Mark McGill, Florian Mathis, Julie Williamson, Mohamed Khamis,
      University of Glasgow
       
    23. Wearable Sensing Technology for Capturing and Sharing Emotional Experience of Running
      Tao Bi,
      University College London
       

    UNIVERSITY RESEARCH GROUPS USING eSense

    60+ academic research groups from all over the globe are contributing to this research effort. We sincerely acknowledge their contributions.

    Contact Us


     
    Pervasive Systems team at Nokia Bell Labs Cambridge maintains this research space voluntarily as a community service.
    Please get in touch with us to contribute to this effort in advancing the earable computing research.

    Credits : Alessandro Montanari, Chulhong Min, Akhil Mathur, Utku Acer, Marc Van den Broeck, Fahim Kawsar