MuSS-Bits
Making music requires a closed feedback loop to evaluate the created with the intended sound. This becomes a challenging task for those with hearing disabilities who are nevertheless interested in learning to play an instrument. Due to limited access to the auditory channel, deaf people have less information to evaluate their performance. The result is a feedback loop-gap.
We developed Music Sensory Substitution (MuSS) Bits, small wearable plug-and-play sensor-display pairs that communicate wirelessly peer-to-peer. The Sensory-Bit captures real-world sounds from various audio sources, extracts rhythm information and sends it to the Display-Bit which provides a visual and vibrotactile representation of the rhythm. The vibrotactile intensity and visual brightness can be adjusted to the user’s need. The decoupling of the display from the sensor part of the sensory-substitution system allows the performer to spatially deploy the Display-Bit, for example on his or her body, provides free limb movement and permits a simple way of selecting the desired audio source. Real-time feedback has been established through onboard processing, nRF communication, and responsive vibration motors.
We believe that MuSS-Bits are a first step towards customizable exploratory music-sensory-substitution systems for deaf people.
PUBLICATIONS
Supporting Rhythm Activities of Deaf Children using Music-Sensory-Substitution Systems.
Petry, B., Illandara, T., Elvitigala, D.S. and Nanayakkara, S.C., 2018, April. Supporting rhythm activities of deaf children using music-sensory-substitution systems. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-10).
Scaffolding the Music Listening and Music Making Experience for the Deaf
Petry, B., Huber, J. and Nanayakkara, S.C., 2018. Scaffolding the Music Listening and Music Making Experience for the Deaf. In Assistive Augmentation (pp. 23-48). Springer, Singapore.
MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds
Petry, B., Illandara, T. and Nanayakkara, S.C., 2016, November. MuSS-bits: sensor-display blocks for deaf people to explore musical sounds. In Proceedings of the 28th Australian Conference on Computer-Human Interaction (pp. 72-80).
MuSS-Bits Provisional Patent
Petry, B., Forero, J.P., Nanayakkara, S.C., 2016, Muss-bits – music-sensory-substitution bits. Singapore Provisional Patent Application: 10201610020P. Filing Date: 29 November 2016.
Ad-Hoc Access to Musical Sound for Deaf Individuals
Petry, B., Illandara, T., Forero, J.P. and Nanayakkara, S.C., 2016, October. Ad-Hoc Access to Musical Sound for Deaf Individuals. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 285-286).