AR piano for improvisation

Project Title: Piano Learning and Improvisation Through Digital Augmentation

The task of learning the piano has been a centuries-old challenge for novices, experts and technologists. Several innovations have been introduced to support proper posture, movement, and motivation, while sight-reading and improvisation remain the least-explored areas. In this PhD, we address this gap by redesigning the piano augmentation as an interactive and adaptive space. Specifically, we will explore how to support learners with adaptive visualisations through a two-pronged approach: (1) by designing adaptive visualisations to learn based on the proficiency of the learner to support regular piano playing and (2) by assisting them with expert annotations projected on the piano to encourage improvisation. We will design and build a piano roll training system integrating these approaches. We will build a model to understand the complexities of learners’ spatiotemporal data and use these to support learning. We will evaluate our approach through user studies enabling practice and improvisation. Our work contributes to how adaptive visualisations can push music instrument learning and support multi-target selection tasks in immersive spaces.

The focus of this research will be on exploring augmented piano systems anchored on the general research question: How can we encourage piano learners to improvise with the help of adaptive visualisations based on spatiotemporal models and expert heuristics? The models will allow us to not only adapt visualisations to individual learners, but also to understand how users use the piano differently. We divide this broad research problem into more specific research questions:

RQ0: What technological interventions have been introduced to support piano learning?
RQ1:How do we build an AR system that teaches
piano improvisation?
RQ2: Does digital augmentation encourage piano learners to improvise?
RQ3: How do we guide the future design of the augmented piano towards a holistic piano learning experience?

In summary, this research attempts the following contributions:

Survey: a survey of the existing landscape of piano augmentations and a list of recommendations to better design them;
Artifact: an augmented piano prototype with improv piano roll;
Theoretical: a pointing model for spatiotemporal data on piano improvisation
Empirical: an understanding on how pianists learn and improvise with adaptive visualizations;

To know more details in this research, we invite you to read our existing publications (see below, PDF-available):

  • Jordan Aiko Deja. 2021. Adaptive Visualisations Using Spatiotemporal and Heuristic Models to Support Piano Learning. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’21). Association for Computing Machinery, New York, NY, USA, 286–290. https://doi.org/10.1145/3450613.3459656PDF
  • Jordan Aiko Deja. 2021. Encouraging Improvisation in Piano Learning Using Adaptive Visualisations and Spatiotemporal Models. In Adjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 24, 1–4. https://doi.org/10.1145/3447527.3474865PDF.
  • Jordan Aiko Deja, Sven Mayer, Klen Čopič Pucihar, and Matjaž Kljun. (2022). The Vision of a Human-Centered PianoarXiv preprint arXiv:2204.06945IMI 2022PDF.
  • Jordan Aiko Deja, Sven Mayer, Klen Čopič Pucihar, and Matjaž Kljun. 2022. A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences? Proc. ACM Hum.-Comput. Interact. 6, ISS, Article 566 (December 2022), 28 pages. https://doi.org/10.1145/3567719PDF. Honorable Mention Paper Award.
  • Jordan Aiko Deja. 2022. Piano Learning and Improvisation through Adaptive Visualisation and Digital Augmentation. In Companion Proceedings of the 2022 Conference on Interactive Surfaces and Spaces (ISS ’22). Association for Computing Machinery, New York, NY, USA, 41–45. https://doi.org/10.1145/3532104.3571464.
  • Jordan Aiko Deja, Sandi Štor, Ilonka Pucihar, Klen Čopič Pucihar, and Matjaž Kljun. 2023. Teach Me How to ImproVISe: Co-Designing an Augmented Piano Training System for Improvisation. In Proceedings of the 8th Human-Computer Interaction Slovenia (HCI SI) Conference.