Score Following is the problem of designing a computational system that is able to follow a musical performance in real time. It has many applications including page turner, automatic accompaniment, lyrics display, etc. One challenge in score following for piano music is the sustained effect, i.e., the waveform of a note lasts longer than what is notated in the score. This can be caused by expressive performing styles such as the legato articulation and the usage of the sustain and the sostenuto pedals, and can also be caused by the reverberation in the recording environment. This effect creates non-notated overlappings between sustained notes and latter notes in the audio. It decreases the audio-score alignment accuracy and robustness of score following systems, and makes them be prone to delay errors, i.e., aligning audio to a score position that is earlier than the correct position.
In this project, we propose to modify the feature representation of the audio to attenuate the sustained effect. We first apply onset detection and treat all frames within a region immediately after an onset as frames that potentially contain the sustained effect. We then analyze the spectra of the signal in these frames and detect spectral components that are considered as an extension from previous notes. We reduce the energy of these components in the audio representation to attenuate the sustained effect. We show that this idea can be applied to both the chromagram and the spectral-peak representations, which are commonly used in score following systems.
We are developing an Android app for a page-turner based on the score following approaches mentioned above. This app will be able to follow a live piano performance in real-time and display the corresponding sections of the score, regardless of the speed and dynamics changes of the performer. Check back later to download the system.
Bochen Li, Zhiyao Duan, An Approach to Score Following for Piano Performances with the Sustained Effect, IEEE Transactions on Audio, Speech, and Language Processing, accepted.
Bochen Li, Zhiyao Duan, Score following for piano performances with sustain-pedal effects, in Proc. International Society for Music Information Retrieval Conference (ISMIR), 2015.