MirrorFugue: Communicating Hand Gesture in Remote Piano Collaboration

Introduction:

The authors of the paper present three interfaces the Shadow, Reflection, and Organ. These interfaces are designed for synchronous, remote collaboration, with focus on remote lessons for beginners. By displaying the to-hand gestures of a teacher on the piano a student can better grasp and understand the necessary skills and hand movements needed to play the piano.

Review:

Throughout the years many companies including Yamaha, Casio and Moog have added extra functions to their keyboards/pianos to help aid in the learning process. There has also been many remote learning platforms introduced to guild the students in leaning the piano. No platform or company however has offered a platform that allows the player to sit at the piano and watch the hand gestures of their teach at hand level. The authors of this paper have offered exactly that. 

Mirror Fugue is designed by using a mirror system. By using wide-angle cameras and projectors (and MAX/MSP and Jitter to manage video and sound) the team were able to deliver 640 x 480 at 30fps video over an ethernet connection from two different locations in the same building. This allowed for a lecture to sit at one keyboard and play, while the student at the other keyboard could watch and mimic the hand gestures and notes played by the lecture. 

The team at the Tangible Media Group have broke their development down into three sections.

Shadow:

This system works by projecting the shadows of  the lecture's hands directly onto the keyboard. In the implementation of this system a video of the lectures hands is projected instead of a silhouette.

Reflection

This system gains its inspiration from the reflections on a lacquered piano. This is where the keys and the players hands are reflected back at them on the surface in front of them. This system allows for the reflection of their remote partner’s hands on directly in front of the keys.

Organ 

  

This system displays an unaltered top-down image of the lectures keyboard. The keys from the projected image are directly lined up with the physical keys on the student's keyboard. 

 

After extensive testing the team concluded that the organ system offered itself as the best learning platform for the students to learn with.

The team plan to keep researching this area with plans to show more of the players body in the projection. 

The MirrorFugure is an exciting new development that in my opinion has a lot of potential to be added into the class rooms of many music schools. It offers a clever system that helps improve the learning of the student.


For more information visit there blog: 

http://web.media.mit.edu/~x_x/mirrorfugue/

There has also been a second paper, MirrorFugue2, which can be found here:

http://tmg-trackr.media.mit.edu:8020/SuperContainer/RawData/Papers/516-MirrorFugue2%20EmbodiedRepresentation%20of/Published/PDF

Also visit http://tangible.media.mit.edu/ to view many more interesting projects from the Tangible Media Group.



Link to paper:

Xiao Xiao and Hiroshi Ishii. 2010. MirrorFugue: communicating hand gesture in remote piano collaboration. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction (TEI '11). ACM, New York, NY, USA, 13-20. DOI=10.1145/1935701.1935705 http://doi.acm.org/10.1145/1935701.1935705