Musical Pitch-Detection and Interactive-Animation: Stars Fall in Music 🌠

Welcome to my undergraduate senior project, a simulating meteor shower animation triggered by pipa (Chinese Lute 琵琶), a traditional Chinese string instrument, through musical pitch detection.

Role

Computing Artist, Programmer, Live Musical Performer

My Contribution

This is a solo project.

Techiniques

MAX 7, MAX Vizzie, Pipa (Chinese Lute 琵琶), Unity3D, Arduino

What is Stars Fall in Music?

It's my undergraduate senior project, a simulating meteor shower animation triggered by pipa (Chinese Lute 琵琶), a traditional Chinese string instrument, through musical pitch detection.

The recording of my live performance was lost. Luckily, this video of my final testing captured the most of it. 😇

Interested in how I brought it live? See below 👇

My pipa (Chinese Lute 琵琶), a traditional Chinese string instrument
A screenshot of my testing performance

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     

Unity3D + Arduino did not work, but Max7 + Max Vizzie did

I initially implemented my idea with Unity3D and Arduino to create a 3D visual effects meteor shower. The static visual effects was nice, however, the buggy importing and exporting connections Unity3D and Arduino was a pain, and the overall pitch control was not smooth. 😔

As a result, I started trying other tools to implement my ideal meteor shower.

1st round tryout
1st round tryout

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

In the 2nd round tryout, I implemented my idea using Max7 and Max Vizzie, both are softwares utilizing iconic java-like programming language to produce visual-audio mixed digital artistic projects. Luckily, I found a way to achieve the pitch detection and control within Max7, which enabled me to play with 3D images interactively controlled by pitch detection technology.

However, the 3D meteor shower did not look as closely like the real meteor shower as I expected. I then tried to work with 2D vector visuals. This time both the pitch detection and the visual effects worked pretty well. With the 3D vector lined visuals, the meteor shower looked more natural and realistic. This is what I was pleased to present! 🥰

2nd tryout — Succeed!
2nd tryout — Succeed!

                                                                                     

Testing the pitch detection before live performance

Before performing live with my Pipa, I did several rounds of testing to make sure everything works fine on the big screen with projector and stereo equipment settings.

                                                                               

It went well! 👏

After my performance, friends and professors went to me saying that they have never seen something amazing like this. I'm glad to have the opportunity to combine my hobby and my work, and I'm glad that people like it. A big THANK YOU to Professor Amy Alexander for her support.❤️ She made me believe that being a computing artist is something cool! 😎