Subscribe
I'm not much of a Flash person and I guess I just hadn't been paying attention since I only found about the new audio features when Scott Schiller added support for them in his SoundManager2 JavaScript/Flash library and later posted a (very cool) favicon VU-meter that would just dance all night to the sound of your music. Since then I've been wanting to do something with those abilities since I figured canvas and live audio data would be a perfect match for some groovy audio visualization.
This is the first of hopefully many such experiments. It's a fun little music video of sorts, where I've just thrown in all sorts of things to the tune of some Radiohead. If you don't care about all the details, scroll down to the end for the link.
So, apparently Flash has a function called
computeSpectrum which returns the current audio state, either as frequency data or as a waveform. In SM2 this has been split into two separate properties of the SMSound object (you have to specifically enable both with a setting and it's only available in Flash 9 mode). The data is available as 256-element arrays of values between 0 and 1. 
For the frequency bars at the bottom, I've simply summed the values in 9 broader groups rather than paint the full spectrum. The same goes for the binary values on the left sides, they're simply averaged in 16 groups and then converted to binary. At the top of the screen you'll see the waveform, drawn at intervals of 8 for performance reasons.
You can pause the whole thing by clicking on the main area. Clicking on the waveform at the top controls the playback position.
Naturally, this is only for
canvas enabled browsers (no Internet Explorer). It runs a lot better for me in Chrome than any other browser. I have not tested it in Firefox 2 but Firefox 3 runs ok but slow (might be better on more powerful machines than mine) as does Opera 10. Safari 4 is fine as well although it stutters a bit at times. I'm sure there's plenty room for more performance improvements but that will have to wait for another time.Another puzzling issue I came across: I get a security error from Flash's
computeSpectrum when trying to run this if I also have a tab open with a YouTube video (probably some other Flash sites as well). Exact error from SM2: "(Flash): computeSpectrum() (waveform data) SecurityError: Error #2122". Happens in at least Firefox and Chrome and a bit of googling tells me it's a Flash issue.I chose Radiohead because I'm a big fan and it just seemed like a nice place to start, given the Flash based visualization of their House of Cards "video" last year. And in case you're wondering, the track is Idioteque off of Kid A, taken from a 2001 performance in Paris that can be seen on YouTube here (just remember the tab issue I just mentioned).
I'm not aware of any other music visualization projects using JavaScript and
canvas as its output medium and Flash as the "backend" (besides simple dancing bars and such), so I'd be very interested if you know of any.Now have fun with the demo!
If you like this, you should definitely check out JuicyDrop, another music visualization project of mine (think Canvas meets Winamp).
20 comments:
Heheh, this is pretty whacky, but the performance seems good. You could port over some AVS effects even ;). See: http://acko.net/design/avs
Very impressive, Jacob. Especially great to see some of the flocking behavior that I've only seen thus far inside the Processing domain.
Rebirth of demoscene!
Awesome mashup :)
Thanks!
@Steven Wittens: The thought had crossed my mind, actually. I've also been looking at MilkDrop, the files for that one would be pretty easy to parse.
http://imgkk.com/
Click JS Demo in the footer. Watch. :P
(Yes, it is partly based on code from that Dynamic Hypnoglow thing)
And yes, I'll admit my algorithm is slightly less complex, but it gets the job done :P
Artistic! One of the niftier uses of SM2's spectrum data I've seen, someone else made a visualizer similar to those seen in WinAmp a number of months back. Unfortunately 20 minutes of searching my email didn't dig it up, but it was pretty good. Maybe they read your blog too and will chime in. ;)
Re: YouTube videos, security errors etc.: With computeSpectrum(), it attempts to read data from all "source" SWFs outputting audio (presumably) at the time, and it can't read "low-level" data by default because of cross-domain security restrictions. Put another way, soundmanager2_flash9.swf on A.com may not access sound data coming from (or originating from?) a .SWF on youtube.com. This may seem a bit odd, but consider that you could potentially "steal" audio from other .SWFs, etc. As far as I know, there is no way to isolate and read only the local .SWF's audio output as the soundMixer object is a Flash system global.
@Darkimmortal: Nice!
@Scott: Thanks for clearing up the security error issue a bit. It had me completely baffled until I somehow figured out that an open YT tab was causing the problem. I guess it makes sense, though, if the sound output isn't tied to the local SWF.
And do let me know if you find that visualizer :-)
OK, after a bunch of email archive and Get Satisfaction searching, I eventually found it. It was a DJ/music project of sorts someone was working on, it came out of a SM2 discussion on frequency vs. waveform. Unfortunately it seems the site is down, at the time of writing at least. The developer is probably still around though and seemed like a cool guy, FWIW.
@Jacob: Neat. I should really look into SoundManager2
@Darkimmortal: Saying " Yes, it is partly based on code from that Dynamic Hypnoglow thing " is kind of an understatement. I don't mind people using part or whole of of my code, especially when it's credited properly unlike here. Interesting use of the Dynamic Hypnoglow thing though.
@Scott: Thanks for digging, too bad it's down. Maybe I'll see if I can contact him.
@Mathieu: Yes you should. Scott has taken all (well, most of) the 'argh' out of sound in the browser, even if you do have to swallow the fact that it's Flash.
Jacob: I met P01 some time in 2002 (via the Ozone Asylum JS/DHTML forum) and while none of us are really big Flash fans, I've come to realize that there are a few areas where it has advantages and is widely available due to the 90-something-percentile install base. That plus ExternalInterface (allowing JS/Flash communication) make doing stuff from JS-land possible, which means it's an attractive potential option to DHTML fans like ourselves.
In the meantime I'd love to see HTML 5 audio and video grow to a point where they can replace the reliance on Flash, and ultimately can provide a similarly full-featured API (eg., allowing access to waveform data and so on.)
Absolutely, I agree. And there is no Flash hate here - the right tool for the right job and all that - there are just some areas where it isn't/shouldn't be necessary to resort to plugin based technology.. Anyway, I suppose its widespread use has both upsides and downsides since it means that we do have that option for things like audio but it also makes it that much harder to replace. So, yea, I love your library but I'm also looking forward to the day that I won't need it!
Hello,
please add your site at http://www.sweebs.com where other people can find you among the best sites on the internet!
Regards
Kris
wow. it's really awesome.. wow~!
Nice. I bet you can't get close to this though! http://www.noisecradle.com
Is it really fair to compare the two, though? Also, check this out for something a bit closer to Noise Cradle.
I've been messing with dong this sort of work in AS3 using FlashDevelop and Flex, it's a lot of fun. I've always been an audio guy, but never actually stuck with it, I've always stuck to programming and game design; working with audio in game design has definitely given me a place to be proud of both game design and audio. I definitely love this kind of stuff though, I'm eager to throw it into game design in a new way. Thanks for sharing, awesome stuff.
Hi,
the links to Scott Schiller and SoundManager2 seem not to work?
Stefan
Links work now. Something has been fixed.
Stefan
Post a Comment