What is Stars fall in music

This is my undergraduate senior project, a simulating meteor shower on a big screen controlled by pipa, a traditional Chinese string instrument, through pitch detection.

When I evoke one specific pitch by playing a specific string on certain position of my pipa, one star will have certain action in corresponding to that specific pitch detected by the system. For example, a star will appear on certain position on the screen, or fall from certain position to certain position on the screen, or change color, or change size, etc. Therefore, with the up and down of a complete music I am going to play using my pipa during the final presentation, a simulating meteor shower will appear on the screen, creating a romantic as well as magic visual feast.


How did I bring it to live

I initially implemented my idea with Unity3D and Arduino to create a 3D visual effects meteor shower. The static visual effects was nice, however, the buggy importing and exporting connections Unity3D and Arduino was a pain, and the overall pitch control was not smooth. As a result, I started trying other tools to implement my ideal meteor shower. 

In my 2nd round tryout, I implemented my idea using Max7 and Max Vizzie, both are softwares utilizing iconic java-like programming language to produce visual-audio mixed digital artistic projects. Luckily, I found a way to achieve the pitch detection and control within Max7, which enabled me to play with 3D images interactively controlled by pitch detection technology. 

However, the 3D meteor shower does not look as closely like the real meteor shower as I expected. I then tried to work with 2D vector visuals. This time both the pitch detection and the visual effects work pretty well. With the 3D vector lined visuals, the meteor shower looks more natural and realistic. This is what I am pleased to achieve! 


Testing the pitch detection

Before perform lively with my Pipa, I did several rounds of testing to make sure the pitch detection works properly, the visuals are the realistic and natural, and everything works fine on the big screen with projector and stereo equipment settings in the proformence room. 

A screen recording of the final testing of 2D vector meteor shower controlled by vice detection within Max7

A screen recording of the final testing of 2D vector meteor shower controlled by vice detection within Max7 on the big screen with stereo equipment settings.


Live Performance

This is a video of my live performance during final presentation of my senior project. The animated meteor shower was controlled by the pitches of the music I played in my pipa. The name of the music is Dance of the Yi (《彝族舞曲》in Chinese), a traditional Chinese dance music. Please see my artistic statement below for the specific reason I chose to play this dance music, and how I got inspired to make a animated meteor shower controlled by pitch detection. 

The live performance of my undergraduate senior project Stars Fall in Music


Why a Meteor Shower Controlled by Music

My Artistic Statement of Starts Fall in Music

My aesthetic goal of this project is to represent the relationship between our mother nature and music through computing and the arts. I got inspiration from a project called The Music of Nature. “All seems well in the world when we hear the uplifting choruses of birds, frogs, and insects, and when we immerse ourselves in the elemental music of wind, rain, thunder, and waves. Not only do these sounds captivate us, they also help us relax and let go of stress, providing us with a respite from our everyday cares and woes. (http://musicofnature.com/ )” I could not agree more with it. From my point of view, the music is a part of natural products and a part of manmade products. It is originally produced by nature, and is later on developed by human beings. As one of the sons of nature, we human are able to directly perceive the music from nature. When we play musics with the musical instruments which are invented by ourselves, we tend to imagine or draw corresponding natural landscapes.

In my project, the pitches of the melody I perform with my pipa trigger a stunning meteor shower. I choose pipa and meteor shower for specific reasons. Although the music is manmade, the musical instrument are made of metal and wood, which are from nature. Although the meteor shower is a simulating digital animation, the idea is draw from the nature. Without the existence of the real meteor shower in nature, it would be impossible for me to create a digital version to imitate the real natural phenomenon. Without the melody created by human beings, the metal which makes the strings and the wood which makes the body of the pipa would never be a magic instrument which produce complicated music that are able to represent various emotion states of human beings.

I choose to play the Dance of the Yi (《彝族舞曲》in Chinese) also for a specific reason. Yi is the name of a minority in China. As a traditional Chinese music created for pipa, Dance of the Yi is supposed to play at night because it tells a story of an outdoor feast of Yi people celebrating harvest with music, dance, and fun activities. It has two parts: the first part is adagio, which shows the preparation of the feast and the solemnly sacrifice procedure; the second part is allegro, which shows the lively dancing and activities during the feast. From my personal experience, Dance of the Yi is a good fitting of the atmosphere created by a meteor shower because of the various atmosphere it brings audiences with. At the beginning of a real meteor shower, stars occasionally fall. Sometimes you have to wait for several minutes to see one meteor pierce the sky. During that time period, the sky is extremely quiet and probably because you are waiting for the next meteor to fall, you are holding your breath and are therefore stressed out. It is very similar to the atmosphere brought by the first part of Dance of the Yi. Soon, more meteors start to fall and the sky looks more hustle and bustle. At that moment, you can feel a strong lively atmosphere for many bright lines flashing on the sky. This atmosphere is very similar to that created by the second part of the music.

Since I found the interesting relationship between the nature and the music I mentioned earlier in this paper, I was trying to figure out how to represent it in an artistic way. Initially, I was considering of representing this relationship through visual music, like the artworks we have studied in ICAM140 and ICAM145A. However, when I looked back to those artworks, I realized that they are only representing simulating visual effects extended by human being’s imagination based on their understanding of music. Those artworks only explored the relationship between music and its visual effects. They have nothing to do with discussing the relationship between music and nature. Because of that, I decided to abandon the idea of visual music, and try something new. After doing several research, I decided to incorporate pitch detection into my project because it makes the project interactive. I got inspiration from this product design called SOUNDVIZ (https://www.soundviz.com/). In this website, the user can upload or record a sound of his or her own voice, and it will be converted into a visual sound wave. The user can customize the shape, color, and style of the sound wave, and choose a print option to transfer it to a fine art print or a gallery wrapped canvas. I think this idea is amazing for it not only turns sound into images, but also allows so much interaction. Traditional visual music just has visual effect to represent sound by changing color and shape; it never allows viewers to interact with the visual effect. This product, however, let users take the initiative by smartly adjust the idea of visual music a little bit. Although users only did a very simple work — recording their voices and upload to the site — they still feel that they are the core of the product because of the uniqueness of the voice of each individual. Here, the key word “uniqueness”, “initiative”, “interaction” let me came up with the idea of pitch detection. I could simply create a simulating meteor shower with a radio tape of Dance of The Yi and makes it visually appealing as well. However, that way the project become static; the viewers have to passively listen to the music and watch the meteors instead of actively interact with it. Although my presentation is not designed for audiences to play my pipa, it is still an interactive project because everyone can trigger the meteor shower by playing my pipa. Also, because different people have different beloved melody to play, the meteor shower would look differently. Thus, everybody can create his or her unique customized meteor shower.