Check out my
new book!
HTML5 Games book
JavaScript + Canvas + SM2 + MilkDrop = JuicyDrop More canvas music visualization - now with 100% more Winamp-iness.

A couple of weeks ago I played around with music visualization using JavaScript/canvas and SoundManager2. Well, I couldn't leave it at that and as I mentioned in the comments, I had an eye on the MilkDrop plugin for Winamp. The result so far is a little Winamp lookalike called JuicyAmp with its own music visualizer JuicyDrop that feeds on Milkdrop preset files.

If you just want to see the pretty colors -> CLICKY. (But please use Chrome or Firefox 3+)

MilkDrop is nice because, although there's a built-in editor in the plugin, the presets are in plain text. They are basically just lists of variables and equations that, with a bit of mangling, can be evaluated as JavaScript. There are also extensive guides that explain how to author presets and how variables are passed around between the different equations. And, even better, the source code for the plugin was released a couple of years ago.

MilkDrop presets consist of a number of different elements (waveforms, shapes, per-pixel effects, etc). Some of them I haven't touched at all, but JuicyDrop supports enough at the moment that a good handful of presets run just fine. That said, there are a whole bunch of problems to work out and the presets included were selected because they looked alright and didn't make it blow up.

I strongly recommend using Chrome for this. Firefox 3 can play too but is probably somewhat slower. There's something screwy going on with Safari, it's like it's refusing to update the display (try holding down a key on your keyboard) (Edit: at least a few people have reported that Safari is working fine for them, YMMV), I'm not sure what exactly is causing it. Opera is a mixed experience for me and it seems to have a problem with playing the music that I haven't found the reason for yet.

The issue with open Flash/sound-using tabs, etc. is of course still here as well (read the comments here for more).

A couple of keyboard controls:
  • Z : switch to smaller (128x128) visualization view (in case of low framerate)
  • X : switch back to normal (256x256) view
  • D : Toggle rendering of deformation mesh points
  • 1 : Toggle basic waveform
  • 2 : Toggle custom waves
  • 3 : Toggle custom shapes
  • 4 : Toggle borders
  • 5 : Toggle per-pixel effects
  • 6 : Toggle video echo
Note that not all presets use all of the features that you can toggle with the keys 1-6.

So without further ado, go have yourself a canvas trip.

I'm not sure where to go from here (besides a lot of optimizing). I'm not done with this, but I'm not sure I want to try to get full MilkDrop support. I believe the presets of today use a lot of pixel shaders anyway, which obviously is no good here. What I might do, though, is add another preset format (probably JSON?) that's a bit easier to work with and then shape that into something more suited for canvas and JavaScript. But for now I have a date with Pixastic..

⇓ 34 comments Darkimmortal

VERY impressive :O

Btw, is Pixastic's editor ever going to be released?

March 24, 2009 at 10:48 AM
Jacob Seidelin

Thanks :D

The editor is what I'm about to start working on. The plan is to make (and release) a new editor that will also be shaped into a Ubiquity command.

March 24, 2009 at 10:57 AM
richtaur

Very cool Jacob! I'm a huge Winamp fan so this gets extra kudos from me :D

March 24, 2009 at 12:57 PM
Arthur Langereis

Cool stuff. Works just fine in Safari 4 beta (Mac version) as well.
Is it too late to send in feature requests for the video and audio element scripting APIs?
Getting frequency and sample data from those elements would in the future obviate flash even more.

March 24, 2009 at 1:09 PM
Jacob Seidelin

Thanks, Matt. Glad you like it :-)

@Arthur: Cool, could just be my Safari that's acting up. And I agree, having those features in the new audio element would be great.

March 24, 2009 at 2:16 PM
Simon

This looks great!


Btw: you'd find lost of takers for it if you ported it into Songbird as an add-on extension!!

March 24, 2009 at 10:29 PM
Stuk

Wow. This is just fantastic! And it runs pretty fast as well (Ubuntu Firefox). Congratulations!

March 25, 2009 at 3:11 AM
mongolito404

It also runs on Midori (WebKit) on Ubuntu (8.10).

March 25, 2009 at 4:59 AM
Dan Kantor

This is the coolest thing I have seen in a while! Great work.

March 25, 2009 at 10:49 AM
Jacob Seidelin

Thanks for the nice comments.

It looks like other people are having better luck with both Safari and Opera than I had. Cool.

March 25, 2009 at 2:08 PM
Scott Schiller

Just for the record, I'm a huge WinAmp fan (I paid the $20 to register and support it back in the day), and it's probably the single app that I miss most about Windows now that I work more on OS X. MilkDrop was a really popular plugin during the WinAmp heyday from what I recall.

March 25, 2009 at 4:22 PM
Elliot

I just javascriptgasmed.

March 25, 2009 at 6:37 PM
Jacob Seidelin

@Scott: A question. You only make one channel of audio data available in waveformData/eqData, right? Any reason for that?

March 26, 2009 at 1:26 AM
virtuelvis.com

For what it's worth: This works great in Opera 10

March 26, 2009 at 2:16 PM
Rovastar

Interesting project. I like what you have done with MilkDrop so far.

I'll be keep an eye out for future versions.

Cheers,

Rovastar
www.milkdrop.co.uk

March 27, 2009 at 9:15 AM
Scott Schiller

@Jacob: Oh yeah, I was initially just doing one channel assuming it'd be faster to pass over ExternalInterface, plus I wasn't sure how much use the feature would get. I'll add left and right with the next version.

March 27, 2009 at 10:10 AM
Jacob Seidelin

@Rovastar: Thanks! :D

@Scott: Cool. I had to hack it in myself for this one. Looking forward to the next version.

March 27, 2009 at 10:36 AM
Gary Johnson

I am trying to get this to run locally. When I run /labs/juicydrop/ from your site it works both in FF release 3.5 beta4 and my debug build. For it to run locally, I am at the point where I am trying to figure out why its not painting the song and visualization and button names on the screen.

I have a few questions that I would like to ask. For example, why do you use setupSoundManager to bring in SoundManager.js? I have 452 milk files but after looking for 10 minutes, I have yet to find one match between mine and the ones in your list in juicyamp.js. When I get to the point of being able to find files, are their limitations on what types of .milk files are supported?

This is awesome.

April 30, 2009 at 9:58 AM
Jacob Seidelin

@Gary: Try setting soundManager.debugMode = true in setupSoundManager() and see what it says. It probably has problems loading SM2 locally and IIRC nothing is done until that happens.

The reason I'm pulling in SM2 after pageload with setupSoundManager() is to get around a Chrome/SM2 bug. Sometimes (seemingly random) Chrome would just not load SM2, doing it this way appears to "fix" it.

The milk files in the list (you can also get them here: http://www.nihilogic.dk/labs/juicydrop/presets/ ) were all found in the packages found here: http://www.milkdrop.co.uk/downloads.htm

And as I said in the post, the presets shown in the demo were handpicked and yes, there are all sorts of limitations, all stemming from the fact that not all of the .milk format is support (and there are also differences in the way JuicyDrop and MilkDrop interprets certain things). And since parts of the .milk file are simply eval()'ed raw, there's also potential for all manners of runtime errors. The best bet is to just throw .milk files at it and see if it renders anything worth looking at.

Hope you'll have fun with it!

April 30, 2009 at 10:19 AM
Adam Schwartz

Hey Jacob,

Using the tips from your email, I was able to hack worldTunes and JuicyDrop together: http://polymath.mit.edu/projects/worldTunes/

For tracks which I'm not hosting localing, I'm using peakData, Math.random(), and Math.sin() in order to have something for waveformData and eqData. It's not great, but it sells the illusion for songs which have a strong beat.

Thanks again for your help and letting me use your code.

Sincerely, Adam

May 22, 2009 at 7:36 PM
Jacob Seidelin

Hey Adam,

Looks great! The app as a whole is just really smooth. Nice job.
If you plan to show her off, might I ask for a link back here from your credits page?

Cheers,
Jacob

May 23, 2009 at 8:06 AM
Adam Schwartz

I don't have a credits page yet, but for now, I added a link on the about page (more or less the same).

I'm still keeping her quiet (I haven't written a blog post yet or anything), but in a few weeks when she hits v1.0, I'll (hopefully) have a nice credits page, help page, documentation page, and an API page for other developers to write plugins and skins.

Which reminds me: at http://editor.pixastic.com/, in the 'Save' dialog, and many others, there's that 'Source' link in the bottom left corner. I understand this is keeping with spirit of open source by helping people see what's going on under the hood and such. But is this feature just for people's curiosity, or is there some bigger-picture-future-API thing at work here? And related to that, what advice might you have for me with open-source-ing / API-ing worldTunes?

May 24, 2009 at 1:12 AM
Jacob Seidelin

That's cool.

The source links in the image editor were just for those interested in how things work behind the scenes. That's about as open source as the editor gets, though, and there is no bigger picture or anything.

Anyway, I think you should definitely get the source out there and making it easy to extend the app sounds great. I'm currently dealing with some of the same issues (or will be soon), since I'm rebuilding the photo editor from scratch.

May 28, 2009 at 12:58 PM
Mark Laff

Problem on FF 3.0.12

Error: uncaught exception: [Exception... "An invalid or illegal string was specified" code: "12" nsresult: "0x8053000c (NS_ERROR_DOM_SYNTAX_ERR)" location: "http://www.nihilogic.dk/labs/juicydrop/juicydrop.js Line: 1378"]

after clicking Play on one of the music selections. Didn't matter which visualization I picked. The music plays but the visualization window is dark.

Perhaps an incompatible update to FF.

July 27, 2009 at 7:40 AM
FRANCIS

WHO THIS READ MY LETTER FRM FRANCIS THANK YOU.

August 10, 2009 at 11:00 AM
FRANCIS

WHO THIS READ MY LETTER FROM FRANCIS THANK YOU YOURS YOURS YOURS.

August 10, 2009 at 11:02 AM
FRANCIS

WHO THIS READ THIS MY LETTER FROM FRANCIS THANK YOU YOURS YOURS YOURS.

August 10, 2009 at 11:04 AM
Greg Kirk

Im pretty sure that canvas uses openGL now (google io had a demo video on youtube of a 3d island using openGL and running in canvas)

So, how about those lovely warp and comp shaders? ;)

February 18, 2010 at 8:54 AM
Jacob Seidelin

Yes, there's both O3D (Google's 3D thing) and the OpenGL-based WebGL for canvas now. Maybe some day. ;-)

February 18, 2010 at 9:05 AM
Anonymous

Very impressive.
Is it free to use your code?

March 17, 2010 at 9:44 PM
createwall

Your Photo to Canvas or Choose from our Gallery! We have a wealth of experience in creating high quality prints for the home and retail market.
thanks
print photos onto canvas

July 12, 2010 at 2:21 AM
Johannes Ziemke

Looks like this question as already asked but is still unanswered: The code is missing any license, could you release it under a open source licence? That would be awesome!

August 14, 2012 at 7:54 AM
xXmicloXx

Yeah, I would need that code too. Can you please add a license?

July 20, 2013 at 1:45 AM
Philippe

Hi Jacob.
Very nice work!
I just tried your JuicyDrop demo on the native Android browser (Android 4.1.2). It works fine! And this makes me sad :)
I'm trying to get peakData and waveFormData from sound.
This works fine with desktop browsers, but with Android browser the peakData and waveFormData arrays are always filled with 0.
And I see your waveFormData array populated as expected!
Do you have any magic wand?
Cheers, Philippe.

July 29, 2013 at 7:40 AM
Post a Comment