web statistics

notes.variogr.am

My name is Brian Whitman. I am a lapsed scientist and sound artist currently co-founder/CTO at The Echo Nest, a music intelligence company in Somerville, MA. As I work on various scaling and media search problems with detours into art projects I'll be posting details here in the hopes that I can learn from others. I'd always like to hear from you if you are working on similar things.

Apr 23rd, 2010 @ 11:10 pm

The Echo Nest Musical Fingerprint (ENMFP)

Tomorrow begins MHD Amsterdam and at it The Echo Nest is releasing a few new things. Some of our engineering team (who deserve a severe callout for all their work, let me stick with their codenames) have been working tirelessly to get “songs” to be a first-class member of our API, and as of today, they are — we now track many millions of songs and you can query for them by name and receive all sort of useful metadata, get similar songs (with amazing results even very deep in the catalog), and even get free (legal) playable audio for a huge collection of major label content (more on this later.) As part of this push to provide data about songs, we have been working on a music fingerprint— a way to resolve an unknown audio file (what we call a “track”) to a large database to identify it in our world (as a “song.”) And we’re ready to release this to the community to see how it performs in the wild.

The design goals of our FP were to base it on Echo Nest audio features, to make it simple to implement and to make it as open as possible. Lock in of content resolution data is a terrible thing, and a large part of The Echo Nest’s focus is to make it easy for people to figure out what their music is about without getting stuck in ID space hell. If you have an iTunes collection and want to automatically make Spotify playlists, we should be able to help you. If you write an app that scans your hard drive for tracks to make great recommendations against MOG or the Limewire store, we should be able to help you. If you want the tempo of every song in someone’s terribly labeled iPod library, we should be able to help you. A fingerprint to us is a utility call— like our search_artists — a way to resolve a music identifier to our set of ID spaces. Echo Nest song IDs, if you choose to use them, give you all of our stuff “for free” — from a single EN SO ID you can get recommendations, artist pictures and bios, blog posts, record reviews, and of course all the audio analysis: the tempo, key, events in the song. But over this year we are rolling in support for any other ID space via Rosetta Stone, so you will be able to return Spotify IDs or get last.fm URLs of the song from the fingerprint. Our goal as always is to be the bridge between music and amazing applications— a platform for music intelligence that lets anyone use any service on any audio to discover and interact with music.

How it works

Our fingerprint is called the Echo Nest Musical Fingerprint (ENMFP) and is based directly on parts of our audio analysis engine that already powers tons of interactive music and music search apps across the globe. We get a detailed understanding of what is happening in a song (note: a song, not just an audio file) for “free” simply by having Tristan be our co-founder, so our work on the ENMFP started there. We worked with audio scientists on ways to scalably hash parts of the analysis and query for “codes” — a sequence of numbers that can match the same song to the ear. We identified an efficient series of transformations of our low level segment description data to make a very accurate code, and our engineering team built a suite of tests, backend servers, and a query API. The ENMFP comes in two parts. The code generator is a binary library that you can compile into your own app. It takes in a buffer of PCM samples (in practice, give it around 20 seconds of 22050Hz mono float PCM), runs a series of signal processing algorithms on the samples, and returns a list of codes. It is as simple as 

    Codegen * pCodegen = new Codegen(_pSamples, _NumberSamples, offset);
    for (uint i=0;i<pCodegen->getNumCodes();i++)
        printf("%ld ", pCodegen->getCodes()[i]);
The server maintains a canonical list of songs with corresponding codes and performs fast lookup. We’ve based the server on some popular open source indexing and storage platforms, and we’ll be releasing our modifications to them as a reference implementation shortly.

Use and open nature

Almost all of this implementation is open. The data behind the server is open by design. Anyone can request full data dumps. Anyone that wants to run their own server can provided that they mirror with the other servers. The only non-normative license is in the code generator, which for now is binary-only, available for most platforms (Windows, Linux 32 & 64-bit, Mac OS X, mobile forthcoming) and free to use in any sort of application — commercial, open source, free, webapp, etc. The only pertinent restriction is that codes are sent to only “authorized servers.” The design of this license ensures that one party does not attempt to usurp the ID resolving space out from under anyone. If The Echo Nest dissolves or gets bought by a large fish cannery on accident, we want to make sure the data and query service live on without us. As a corollary, we don’t want anyone “hiding” new resolved tracks from the ID space. Anyone that collects new songs via this fingerprint has to share their data, plain and simple. This hopefully ensures that over the years the combined knowledge from all uses of the ENMFP will catalog every single piece of music available on the internet, and the data will be available to all. We want the ENMFP to grow into a public internet utility.

Features

Anti-features

How to use

First, you need an Echo Nest developer API key if you don’t already have one. Next, familiarize yourself with the alpha_identify_song API. (As of right now, before we release the server source, the Echo Nest is hosting the only query server via this API.) There is instructions there on how to receive the libcodegen binaries. The libcodegen package also ships with an example code generator that you can call from the commandline, so no worries if you aren’t ready to do some compiling.

How to help

We see the ENMFP as a community project just getting started. If you are interested in booting your own mirror server, or if you have experience with FP tasks, want to help with QA, automated testing, have a large catalog to ingest or test against, please get in touch.

we are especially grateful for the work of Unrepentant Nagios Installer (UNI), Guy Who Fights With Me About the Word “Track” Every Fucking Day (GWFWMAWTEFD), Drinks Turret Coolant (DTC), Mr. HTML5 Canvas 2010 (HC2), So-Glad-I-Kept-You-Out-Of-The-Media-Lab (SGIKYOOTML), Skinny Tie (ST), Main Ontology Offender (MOO), Future Performable Employee (FPE), and of course Ben Lacker (BL)

Feb 7th, 2010 @ 6:18 pm

Jan 29th, 2010 @ 12:45 pm

Primer on new Echo Nest search_tracks, capsule, and get_analysis APIs

Note: if you are interested in these APIs know that they are no longer alpha — see our new Song APIs for the latest information. At Stockholm Hack Day we’re announcing three or four new APIs that are going to stay in our “alpha” sandbox for now. These are officially unsupported but we will work with anyone who has a use case for them. For now, the instructions will stay here until we promote them to production APIs.

For all of these alpha APIs you will need a developer key from The Developer Nest. I will use “YOUR_KEY” as the key in the examples, make sure to replace this or none of the calls will work. We reserve the right to pull or throttle access to alpha APIs at a different rate from the supported ones. Please be warned that these are not production ready, we will be making enhancements and restarting servers, there will be guaranteed downtime. If you are interested in building a product with these new APIs please contact us..

capsule

What: returns a rendered mp3 and swf url given an XSPF playlist that is composed of a “megamix” or “jam” of the included tracks in order.

How:

http://developer.echonest.com/api/alpha_capsule?api_key=YOUR_KEY&xspf_url=http://path_to.com/playlist.xspf

Returns

{'flash_url': 'http://thisismyjam.com/flash/jam.swf?api_PNWUcuxGJ5/1264353117.xml', 
'tag': 'api_PNWUcuxGJ5', 
'mp3_url': 'http://echonest-capsule.s3.amazonaws.com/api_PNWUcuxGJ5/1264353117.mp3'}

Parameters:

Returns:

Notes:

get_analysis

 DEPRECATED 4/23/10 — GO HERE

What: returns the entire analysis for an Echo Nest track ID. (more ID types to come soon.) Saves multiple calls (get_segments, get_tempo, etc etc) and allows you to get the analysis of tracks you did not upload (after finding them with search_tracks.)

How:

DEPRECATED 4/23/10 — use songs/search with the audio_summary bucket to get the full analysis for a song 
http://developer.echonest.com/api/alpha_get_analysis?api_key=YOUR_KEY&trackID=TRMAOUK1254889A145

Returns

{"status": "OK", "analysis": 
	{"bars": [{"duration": 0.75963000000000003, "start": 0.37046000000000001, 
	...

Parameters:

Returns:

Notes:

search_tracks

 DEPRECATED 4/23/10 — use songs/search for this

What: Given a number of possible query types return a list of matching tracks that The Echo Nest maintains. Search on metadata (artist, title), audio data (tempo, duration, loudness, key, mode, etc), cultural data (automatically computed tags, aka “topterms”, familiarity and hotttnesss, collated edited style and genre classifications, etc.) and even location data (latitude / longitude of artist origin.)

How:

DEPRECATED 4/23/10 — GO HERE 
Look for a song:
http://developer.echonest.com/api/alpha_search_tracks?api_key=YOUR_KEY&title=My%20Sound&artist=Squarepusher

Get the loudest romantic song:
http://developer.echonest.com/api/alpha_search_tracks?api_key=YOUR_KEY&query=romantic&sort=-loudness&results=1

Get the songs between two tempos:
http://developer.echonest.com/api/alpha_search_tracks?api_key= YOUR_KEY&constraint_tempo_min=120&constraint_tempo_max=125&artist=Squarepusher


Returns
{"status": "ok", "results": 
	[{"trackID": "TRMAOUK1254889A145", "title": "Dream On", "artist": "Aerosmith"},
	...

Parameters:

Returns:

Notes:

identify_track

 DEPRECATED 4/23/10 — use songs/alpha_identify_song

What: Returns metadata for a track given Echo Nest Musical Fingerprint hash codes.

Notes:

Jan 25th, 2010 @ 10:32 am

Moonbelly

Geocoded place names mentioned in the entirety of “40 Stories”, via Google Books, some sort of future library that no one talks about as we are too scared of it:

Moonbelly sang a new song called “The System Cannot Withstand Close Scrutiny.”

The system cannot withstand close scrutiny
The system cannot withstand close scrutiny
The system cannot withstand close scrutiny
The system cannot withstand close scrutiny
Etc.

—City Life, Donald Barthelme

Dec 13th, 2009 @ 6:19 pm

KFW’s excellent “Modena” video.

Dec 6th, 2009 @ 5:43 pm

(via meaninglesssyllables)

RIP Jack Rose, 1971-2009. Please, if you can, put on “Kensington Blues” for me.

Reblogged from a-ne-na.

@ 1:36 pm

I hate you, internet

I hate you, internet

Dec 3rd, 2009 @ 5:15 pm

The Echo Nest Remix API talk at Dorkbot NYC, March 4th 2009

About nine months ago I gave a funny talk with my main man Douglas at his world-dominating Dorkbot NYC about Echo Nest Remix. Slideshare’s not so good with the audio part of things so it may be kinda dry, but here it is:

The Echo Nest Remix at Dorkbot NYC, March 4 2009

I remembered to post it because Paul made something hilarious with it today in just a few lines of code. I really think Remix should get more love; I’m on it, and some new Echo Nest features are coming out that will make it even more interesting.

Nov 29th, 2009 @ 10:33 pm

A Singular Christmas (2004)

Five years ago today I released “A Singular Christmas,” the rendered output of a piece of software that listened to hundreds of Christmas songs and tried to compose its own new holiday standards. It ended up as my most successful thing ever by a few orders of magnitude: 600,000 people downloaded it over the space of three weeks. I was on BBC Radio on Christmas Eve; Wired Magazine tried to arrange a photograph of me at a club performing on top of a rack of servers; Pitchfork reviewed it well. Quite parenthetically, “A Singular Christmas” has been to date the last major piece of music I worked on: the weird excitement behind it was my main driver for turning down “safe” jobs to start The Echo Nest with Tristan a few months later, and that enterprise has changed my priorities in ways I couldn’t have predicted.

While I’m sure the vast majority of those hundreds of thousands heard just the first few seconds of the first song before closing their browser, I quite liked “A Singular Christmas” (I can say this because the computer made it, not me!) and each holiday season I’ll get a few nice emails from likeminded people reminiscing about it and asking where to listen to it. As I was a PhD candidate at the MIT Media Lab when I made the record, every single year since the release— in between board meetings and key-value store rebuilds or whatever the hell it is I do with myself these days— I’ve had to ask a current student to find the computer that it was hosted on and plug it back in as a new consortium or robot displaces the latest server closet there constantly. But I’ve had it with self-hosting, this year I am going to the cloud — my blog here at Tumblr can take care of the adjoining text while my good friends at SoundCloud are graciously hosting the audio. Maybe it’ll stay up for more than a week.

Enjoy, happy holidays, please write, xo, -b

Listen and download

A Singular Christmas by bwhitman

If you want to download the whole thing as a .zip file, here you are, it’s 61MB.

Statement and Methodology

Holiday music is the first broadcast of the season. Much like the sudden alert of spring birds, the forces that schedule, produce and filter Christmas hymns from back catalogues and metal shelves in storage lockers out to 8 inch mono white grilled speakers beamformed onto precisely calculated endcaps are a fantastic mystery that is never asked to reveal itself. The truth is surely offensive, some pressed shirt with a copy of SPSS pastes the predicted launch dates and harmonies into a memo and forwards it to General Marketing. I’ve seen their mood circumplex for every chromatic jingle plotted over revenue– it’s in a Microsoft Office 95 file, the fonts look awful and only prints on legal paper. Rest assured that they’ve ran the numbers– they have the data. It’s the only genre that lets musicologists and A&R guys sit in the same room.

As a result, whether we participate or not, we are trained to associate the march from car to heated revolving door, from family pie to adjoint family pie, from ghost tree stand caravans to footprints of dry brown needles leading to the trash in January with the twee pasture of treble and tines and abrupt key modulation, led along on our heartstrings by wooden horses. X, you, times n, a projection through some unknown (until now) auditory stimulus, equals Y, a fond memory of everyone in a sweater. X and Y jitter irretrievably and tragically over the years, but n, static and full bandwidth, holds them together. I always liked to think of every sound, every instrument, every vocal phoneme and every delay tail in Phil Spector’s classic “A Christmas Gift For You” as some hidden variable linking some version of me to some imperceptible holiday memory. Played as it is, it’s every holiday I’ve had and still holidays I’ve yet to experience.

So then what is special about Christmas music? Let’s take the nativist view– that there is something in the composition, construction, timbre or production in every popular Christmas song that makes it fit into the genre. Some predefined, baked in, Chomsky grammar style language of melodies and instruments. So play a Christmas song to someone who’s never experienced a Christmas before. What do they feel? Do they rush out and buy spray-on snow? I never got around to doing the study. What I could do is try to distill holiday music down to its barest essentials. My hypothesis was that if we could figure out the dominant components of Christmas music, and train a system looking only at the audio to make more of it from those components, and if that new music passes the Turing test of the general public considering it Christmas music, then yes, we’ve cracked the code — we can have Holiday Forever, a Singular Christmas.


Sheba and Kristie in front of the rack of servers used to render A Singular Christmas.
MIT Media Lab, November 2004

The recipe for generating this Eigenmusic (”synthesized music representing the maximal variety of the input music”) was cooked up in 2003 as the live radio station “Eigenradio.” It was a hilarious joke if you laughed at beehives, pleasing if you liked electric closets. Here’s what the process is: you parameterize music into some set of features (pitch content, frequency response, high level structure, etc) and set some rate — say a set of features every 100 milliseconds or four beats. You then pass those series of features to a popular statistical algorithm that tries to remove dependence among variables in the feature– removing “redundant” information– perceptual compression. Repetitive structures such as beat and harmony are whittled down to a representation that can always expand back later.

With this new compressed representation we have some powerful new tools. We can reduce two songs and see how close their representations are– since we’ve removed some unimportant noisy stuff it works better than comparing the whole slow song. We can also tweak the representation and play it back again. Dehydrate a tune into two numbers, you’ll probably get a measure of loudness and some measure of the most dominant low frequency, these are now synthesizer knobs you can tweak and compose around.

The fun stuff happens when you take more than one song, compress them all as a unit, and then re-create the original again, play it back. What you’ve got is the computer trying to spin around all the things it heard that it thought were important about lots of music. But always remember– what a computer thinks is redundant are the very things we rely on musical enjoyment- repetition, patterns, harmonies, beat — and what it thinks are important are things we would never want to listen to alone.

So A Singular Christmas is the reduction of dozens of holiday songs, from grim coffee shop collections to poorly recorded indie one-offs. Dozens of holiday records went into the machine, and out came the sixteen tracks you can hear today. We synthesized those tracks by randomly permuting those tweakable knobs over two weeks of sixteen computers blithely ignoring any real work; each one rendered about a dozen New Holiday Classics. As for the role of composer, maybe semi-conductor– I locked myself and family up in a darkened studio on the top floor at MIT, we turned up the speakers, played every track in a row, and I hovered my index finger over the Delete key. There’s not a single window in that place, but I’m sure it started snowing.

— September 2006, Somerville MA

Historical errata

A Singular Christmas (ASC) started as an experiment meant to go in my dissertation to see if a computer could determine automatically if a song is a holiday song. It’s actually a very hard task and I really hope MIREX takes it up sometime. Like most things I gave up on the science part once I realized how pretty the inversions were.

The original thank-you line on ASC’s webpage is gone, but I bet it said: “Kelly Dobson, Victor Adan, Barry Vercoe, Keith Fullerton Whitman, Dan Ellis.” Dan especially, as his MQ-resynthesis matlab code was crucial to the enterprise. KFW provided a large % of the sample bank known as “all Possible Sounds” that ASC rendered from; Kelly was a big help with editing, sequencing and general art inspiration; Barry of course as my advisor foot the institutional “bill” for the computers and is somewhat a personal hero of mine; Victor I am sure paced around a lot and told me how much better it could have been if I embodied Schenkerian analysis into the algorithm.

My personal favorite on this record is the last track, “Holy Night,” which is a bit different from the rest. As a test I only trained it on versions of “Silent Night” in the collection. The output— sort of a 90 second vamp on three notes in a verse with the “orchestra” spinning around it trying to anneal in— is to me everything that is beautiful about this sort of automatic music.

Nov 24th, 2009 @ 3:21 pm

Music Hack Day Boston

Last weekend I was happy to be a part of Music Hack Day Boston. The Echo Nest co-organized it and sponsored it; I sat on a panel, helped with some “local coordination,” put up some awesome people at my place nearby, and generally did everything but make a hack myself. (I tried, really.)

Below is an unordered list of “things I learned” last weekend.

Elissa is a force of nature

While everyone has already thanked the amazing Paul Lamere from the Echo Nest, Jon Pierce from Betahouse / Awesome Foundation and Dave Haynes from Soundcloud for their organization help, I need to publicly declare on the entire internets that while I’m really happy for them and i’ma let them finish, but Elissa Barrett (Echo Nest’s marketing / director of all things) was the best music hack day organizer of all time. Just because she doesn’t tweet or get up all in the social webs doesn’t mean we can’t give her props for all the work she did— from booking all the bands to wrangling the scheduling to taking the thousands of boxes of uneaten sandwiches to the homeless shelter she really made Boston MHD possible.

last.fm are awesome

Seriously, it was hell of inspiring watching those dudes all get on so well after such a long time even after a few are moving on to awesome things. I can only hope that in 4 years I can still make fun of Paul for his terrible puns without him trying to deck me. Matt, David, James, RJ, toby, etc.. all serious “super music” players and they made it work because they believed in it. Much respect.

Playdar is going to be something

Playdar, “the content resolver with a series of questions” was a huge presence at the MHD, starting with a “Playdar Summit” at the Echo Nest office on Friday, which kind of went like this:

The whole crew was there, J Herskowitz, Toby, RJ, the Twones guys, James, a few EN guys, the two Lucases, Dan from AOL… The Echo Nest is getting involved in Playdar in some hopefully useful way very soon, it obviously fits in our general attitude on how music should get found and shared and thought about.

The Echo Nest needs to do a better job explaining itself

17 of the 41 hack day projects used Echo Nest APIs, including all 3 top winners, almost all of them from our Analyze product. We do a lot @ EN, perhaps a little too much. A couple of projects used our (awesome) get_similar artists call, and one used our new get_images call. But no one even thought about or touched our feeds (get_reviews, get_blogs) stuff, which probably constitutes 60% of EN engineering time. Of course, we sell that stuff to non-hack-day companies and the text in those documents are the basis of a lot of our recommender / similarity results, but still… if the future of music is sitting in this room and they don’t want to make an app that uses our crawl data, we should think a bit about what that means.

Automatic music is going to be big next year

My old colleague Rif did an excellent a-from-b style remix using some math that sounded quite pleasant. Rob O won the entire hack day with his outlierfm.com project.

I briefly mentioned this on the panel I sat on, but music remixing / automated mashup / easy creation tools are going to start popping up a lot more. EN is obviously invested in this via Remix and it was good to see similar things like Indaba and Aviary appear at MHD, but are all still a relatively dorky proposition. The first easy to use auto-remixer that is not designed by a crazy person will be amazing.

Hack Days need to be technology-focused

I speak for myself, but MHD is not MIDEM, it’s not the Pho list or SXSW, it’s not some PRO-sponsored junket where we talk about rights issues. It’s a bunch of excited people trying to code out new ways to get people excited about music. We can deal with the monetization, licensing and business models later. This makes me a bit of a hypocrite since I sat on one but the less panels at the next one, the better. More time for API workshops, soldering, meeting people, less time listening to (interesting) people delivering opinions to an assembled crowd.

Of course no one was forced to go to the panels and I do appreciate that the panels bring in some non-hacker types in the audience to even things out but perhaps a good model for the next MHD is to do these “business things” (lightning talks, panels, keynote sessions) on a completely different day from the hack day(s).

Anthony slightly touched on this in his blog post — the Boston MHD people did a great job here but I think it should go even farther towards technology.

Late edit because I’ve already gotten two DMs about “why I hated the panels” — I didn’t, I thought they were great (especially Chris Dahlen’s moderation of the discovery panel and Derek’s amazing comments during the music biz one.) But I don’t think they belong in a scheduled session during a hack day. The urge to see these great people is too strong and gets in the way of doing stuff. Schedule them after final demos for the next one, everyone is happy.

My favorite “hacks”

Archive · RSS · Theme by Novembird