A system based on Microsoft’s Xbox Kinect has been developed that aims to help children with autism and/or profound and multiple learning disabilities (PMLD) to take part a more active part in music lessons.
The Kinect Audio Project (KAP), which is aimed at users with autism spectrum conditions, mobility, learning or speech disabilities is intended to encourage exercise, understanding of movement and association, and aid independent participation.
Based on audio-visual cues, KAP displays a camera image of the user and replicates the user’s hand movements on-screen through oversized, animated gloved hands. The user can move their hands to interact with on-screen visuals, which in turn trigger sounds.
It has been developed by Patricia Afari, an MSc Computing student at Goldsmiths, University of London. “Children with profound learning difficulties are unable to perform typical activities in the usual way,” she explained.” By enabling children to interact and recognise their own movements, we hope to encourage independence.
“KAP enhances children’s own abilities and through engagement with the technology they can better communicate with teachers and other pupils.
“Children learn through repetition, so it was important not to incorporate randomised audio responses. Instead, each visual has been assigned a particular note or instrument.”
KAP incorporates hardware and software from gaming technologies, namely Microsoft’s Kinect controller and Firelight Technologies’ FMOD Ex Programmer’s API. It is currently being trialled at South Downs Community Special School, Eastbourne under the supervision of Tom Smurthwaite, an interaction designer who has 12 years’ experience working with people with PMLD.
“Children with mobility problems often fall at the first hurdle of playing a musical instrument, being unable to hold or operate it,” Smurthwaite explained. “Gesture recognition gives them the opportunity to join in. Being based on budget consumer technology and free downloadable software, KAP has the potential to be a required tool in special education. “
