Musical Pitch-Detection and Interactive-Animation: Stars Fall in Music 🌠

What is Stars Fall in Music?

It's my undergraduate senior project, a simulating meteor shower animation triggered by pipa (Chinese Lute 琵琶), a traditional Chinese string instrument, through musical pitch detection.

‍

The recording of my live performance was lost. Luckily, this video of my final testing captured the most of it. 😇

‍

‍

Interested in how I brought it live? See below 👇

‍

‍

Unity3D + Arduino did not work, but Max7 + Max Vizzie did

I initially implemented my idea with Unity3D and Arduino to create a 3D visual effects meteor shower. The static visual effects was nice, however, the buggy importing and exporting connections Unity3D and Arduino was a pain, and the overall pitch control was not smooth. 😔

As a result, I started trying other tools to implement my ideal meteor shower.

‍

‍

In the 2nd round tryout, I implemented my idea using Max7 and Max Vizzie, both are softwares utilizing iconic java-like programming language to produce visual-audio mixed digital artistic projects. Luckily, I found a way to achieve the pitch detection and control within Max7, which enabled me to play with 3D images interactively controlled by pitch detection technology.

However, the 3D meteor shower did not look as closely like the real meteor shower as I expected. I then tried to work with 2D vector visuals. This time both the pitch detection and the visual effects worked pretty well. With the 3D vector lined visuals, the meteor shower looked more natural and realistic. This is what I was pleased to present! 🥰

‍

‍

‍

Testing the pitch detection before live performance

Before performing live with my Pipa, I did several rounds of testing to make sure everything works fine on the big screen with projector and stereo equipment settings.

‍

‍

‍

It went well! 👏

After my performance, friends and professors went to me saying that they have never seen something amazing like this. I'm glad to have the opportunity to combine my hobby and my work, and I'm glad that people like it. A big THANK YOU to Professor Amy Alexander for her support.❤️ She made me believe that being a computing artist is something cool! 😎

‍