file.txt

For our final project, we wanted to find a way of visually representing music. Music is important to almost everyone, and making a visual interpretation of it seemed a fun and creative exercise in engineering. We decided to make our project using Javascript, with the eventual goal of hosting the resulting program online. Additionally, it would allow us to be become more familiar with WebGL.

The first primary challenge of this project was figuring out how to process each mp3 file. We used javascript's WebAudioAPI to create a computation graph that splits the music into its left and right channels and then runs sequential Fourier transforms on the raw music data.

These frequency components are then used to guide our visualizations. We created two different variants of the fire simulation by dividing the 128 total frequency bins generated by the DFT among each set of fire pillars differently. The first variant splits them evenly, with each pillar representing a group of four consecutive bins. For both variants, the bins on the left represent the left audio channels and the bins on the right represent the right audio channels. In each channel, the lowest frequencies are located in the middle of the screen and the highest frequencies are near the edge. For the first visualization, we modified color, size, height and speed of each of these pillars as a function of magnitudes. In the second variant, the color is now a function of frequency rather than magnitude and the high frequencies near the sides of the screen are slightly amplified due to a different binning algorithm. Here are the results.

Our next visualization takes these fft values and maps them onto a sphere. We wanted to a dynamic visual interpretation of music on a 3 dimensional object. We made a total of 3 visualizations. Each visualization scales the vertices of the sphere along its normal according to some function of the fft. The first visualization we made sort of looks like a spiral - to create it, we mapped fft values to various vertices along the sphere and interpolated values between them. The next visualization edits the rings of the sphere - in this case we downsampled the fft to the number of horizontal rings of the sphere and scaled each ring by its corresponding magnitude. we wanted the the last visualization to resemble a pair of wings on the sphere. We again downsampled the fft but scaled vertices only along a single axis to generate this effect. In addition, the user can toggle whether or not to render the sphere and control the sphere with mouse controls.

We really enjoyed this project Thanks so much for watching.