Brian Whitman @ variogr.am

‘Hungry, So Angry’ by Medium MediumThe TV thing’s blue,...

Tue, 29 Jan 2013 10:53:23 -0500

‘Hungry, So Angry’ by Medium Medium
The TV thing’s blue, it’s been finished for hours

‘I Was Made To Love Her’ by Stevie WonderLike a sweet magnolia...

Fri, 25 Jan 2013 15:33:41 -0500

‘I Was Made To Love Her’ by Stevie Wonder
Like a sweet magnolia tree

linesdotscircles: seventy + eight

Thu, 24 Jan 2013 10:45:11 -0500

linesdotscircles:

seventy + eight

backjammon: bwhitman’s Kurt Weill & Ira Gershwin jam...

Tue, 15 Jan 2013 12:28:52 -0500

backjammon:

bwhitman’s Kurt Weill & Ira Gershwin jam gets Zeng Fanzhi’s “This Land so Rich in Beauty No. 2” [2010] as its background. Special.

3G Wireless in Ubuntu on your ARM Chromebook

Mon, 31 Dec 2012 10:29:58 -0500

If you have an ARM Chromebook with 3G access and want to put Ubuntu on it and also keep your 3G access, do the following:

If you haven’t already, set up 3G in ChromeOS and make sure it works.
Install Ubuntu on an external drive or SD card
sudo apt-get install wvdial
Put this in your /etc/wvdial.conf after replacing the XXXs with your phone number that Verizon set up for you. You should have an email from them when you activated their free 100MB/month plan with the number in it.
Then run sudo wvdial.
You may have it do it a couple of times before you see pppd do stuff. You can exit out of wvdial, pppd stays running until you kill it.
I so far cannot get the network configuration GUI to play nice with this, but I’ll let you know if I do.

How music recommendation works -- and doesn't work

Tue, 11 Dec 2012 11:56:00 -0500

When you see an automated music recommendation do you assume that some stupid computer program was trying to trick you into something? It’s often what it feels like – with what little context you get with a suggestion on top of the postmodern insanity of a computer understanding how should you feel about music – and of course sometimes you actually are being tricked.

Amazon’s recommendations for Abbey Road

No one is just learning that if they like a Beatles album, they may also like five others. Amazon is not optimizing for the noble work of raising independent artists’ profiles to the public, and they’re definitely not optimizing for a good musical experience. They’re statistically optimizing to make more money, to sell you more things. Luckily this is the fruit fly of music recommendation, the late night infomercial quality of a music discovery experience that also might dry your lettuce if you spin it fast enough. And I doubt Amazon would ever claim otherwise.

The rest of the field has gotten pretty far since then and we’ve now got tons of ways to discover music using actual qualities of the music or social cues of what your friends are listening to. But I still hate seeing examples like the above. I hate thinking there are forces at work in music discovery that don’t have listeners’ best interests at heart and I want to make them better. I want to walk through all the ways music recommenders work or don’t, and concentrate of course on the one I know best – The Echo Nest’s – which you’re probably using even if you don’t already know it. And most importantly, I want to talk about what we can do next.

Before I get into it, a brief history of who I am: I’ve been working on music recommenders and music retrieval since 1999, academically and in industry. In 2005 I started The Echo Nest with my co-founder Tristan. We power most online music services’ discovery using a very interesting series of algorithms that is sort of the Voltron-figure of our two dissertations and the hard work of our 50 employees in Boston, SF, NYC and London. And we’ve been on a bit of a tear – just in the past year alone we’ve announced that we’re powering music discovery features for eMusic, Twitter, EMI, iHeartRadio, Rdio, Spotify, VEVO and Nokia – with some new heavy hitters yet announced – to add to our existing customer base that includes MTV, MOG, and the BBC. And through our API we have tens of thousands of developers making independent apps like Discovr, KCRW, Muzine, Raditaz, Swarm, SpotON and hundreds more.

We’ve been a quiet company for a while and with all this great news comes a lot of new confusion about what we do and how it compares to other technologies. Journalists like to pin us as the “machine” approach to understanding music next to the “human” of our nearest corollary (not competitor) in the space – Pandora. This is somewhat unfair and belies the complexity of the problem. Yes, we use computer programs to help manage the mountains of music data, but so does everyone, and the way we get and use that data is just as human as anything else out there.

I’ll go into technologies like collaborative filtering, automatic content based recommendation, and manual approaches used by Pandora or All Music Guide (Rovi). I’ll show that no matter what the computational approach ends up being, the source data – how it knows about music – is the most important asset in creating a reliable useful music discovery service.

What is recommendation? What is it good for?

Musicians are competing for an audience among millions of others trying just as hard. And it’s not the listener’s fault if they miss out on something that will change their lives – these days, anyone can gain access to a library of over 15 million songs on demand for free. To a musician turned computer scientist (as I and so many of my colleagues are) this is the ultimate hidden variable problem. If there was something “intelligent” that could predict a song or artist to a person, both sides (musician and listener) win, music is amazing, there’s a ton of data, and it’s very far from solved.

But anyone in the entire field of music technology has to treat music discovery with respect: it’s not about the revenue of the content owner, it’s not about the technology, it’s not about click through rates, listening hours or conversion. The past few years have shown us over and over that filters and guides are invaluable for music itself to coexist with the new ways of getting at it. We track over 2 million artists now – I estimate there are truly 50 million, most of them currently active. Every single one of them deserves a chance to get their art heard. And while we can laugh when Amazon suggests you put a Norah Jones CD in your cart after you buy a leaf blower, the millions of people that idly put on Pandora at work and get excited about a new band they’ve never heard deserve a careful look. Recommendation technology is powering the new radio and we have a chance to make it valuable for more than just the top 5 percent of musicians.

When people talk about “music recommendation” or “music discovery” they usually mean one of a few things:

Artist or song similarity: an anonymous list of similar items to your query. You can see this on almost any music service. Without any context, this is just a suggestion of what other artists or songs are similar to the one you are looking at. Formally, this is not truly a recommendation as there is no user model involved (although since a query took the user to the list, I still call these a recommendation. It’s a recommendation in the sense that a web search result is.)
Personalized recommendation: Given a “user model” (your activity on a service – plays, skips, ratings, purchases) a list of songs or artists that the service does not think you know about yet that fits your profile.
Playlist generation: Most consumers of music discovery are using some form of playlist generation. This is different from the above two in that they receive a list of items in some order (usually meant to be listened to at the time.) The playlist can be personalized (from a user model) or not, and it can be within catalog (your own music, ala iTunes Genius’s or Google’s Instant Mix playlists) or not (Pandora, Spotify’s or Rdio’s radio, iHeartRadio.) The playlist should vary artists and types of songs as it progresses, and many rely on some form of steering or feedback (thumbs up, skips, etc.)

Where popular services sit in discovery
	Personalized	Anonymous
Playlist	Pandora	Rdio radio
Suggestions	Amazon	All Music Guide

These are three very different ways of doing music discovery, but for every technology and approach I know of, they are simply applications on top of the core data presented in different ways. For example, at the Echo Nest we do quite a bit to make our playlists “radio-like” using our observed statistics, acoustic features and a lot of QA but that sort of work is outside the scope of this article – all three of our similarity API, taste profiles (personalized recommendation) API and playlist API start with the same knowledge base culled from acoustic and text analysis of music.

However, the application means a lot to the listener. People seem to love playlists and radio-style experiences, even if the data driving both that and the boring list of songs to check out are the same. One of the great things about working at the Echo Nest is seeing the amazing user interfaces and experiences and people put on top of our data. Listeners want to hear music, and they want to trust the service and have fun doing it. And conversely, a Pandora completely powered by Echo Nest data would feel the same to users but would have far better scale and results and thus add to the experience. Because of this very welcome sharding of discovery applications, it’s helpful less to talk about these applications directly and more to talk about “what the services know” about music – how they got to the result that Kreayshawn and Uffie are similar, no matter where it appeared in the radio station or suggestion or what user model led them there. We can leave the application and experience layer to another lengthy blog post.

My (highly educated, but please know I have no direct inside information except for Echo Nest of course) guesses on the data sources are:

How popular services know about music
Service	Source of data
Pandora	Musicologists take surveys
Songza	Editors or music fans make playlists
Last.fm	Activity data, tags on artists and songs, acoustic analysis[1]
All music guide	Music editors & writers
Amazon	Purchase & browsing history
iTunes Genius	Purchase data, activity data from iTunes[2]
Echo Nest	Acoustic analysis, text analysis

There are many other discovery platforms but this list covers the widest swath of approaches. Many services you interact with use either these platforms directly (Last.fm, Echo Nest, AMG all license data or give away data through APIs) or use similar enough approaches that it’d be not worth going into in detail.

From this list we’re left with a few major music knowledge approaches: (1) activity data, (2) critical or editorial review, (3) acoustic analysis, and (4) text analysis.

The former two are self-describing: you can learn about music by the activity around it (listens, plays, purchases) – Kreayshawn and Uffie are considered similar if the same people buy their singles or rate them highly – or you can learn about music by critical review, what humans have been pretty good at for some time. Of course, encoding activity (via collaborative filtering or taste mining) and critical review (via surveys or direct entry) into a database is a relatively recent art.

The latter two, acoustic and textual analysis, were developed by the field as a reaction to the failures of the first two. I’ll go into much greater detail on those as it’s how Echo Nest does its magic.

Care and Scale

The dominating principle of the Echo Nest discovery approach from day one has been “care and scale.” When Tristan and I started the company in 2005 we were two guys with fresh PhDs on music analysis and some pretty good technological solutions; Tristan’s in the acoustic analysis realm (a computer taking a signal and making sense of it) and mine in the data mining and language analysis space (understanding what people are saying and doing with music.) We surveyed the landscape at the time for discovery and found that almost every one suffered from either a lack of care or scale, sometimes (and often) both. The entire impetus of doing a startup (not an easy choice for two scientists and anyone that has met us knows we are not the “startup type”) was that we thought we had something between the two of us that could fix those two problems.

Care & Scale

Scale is easy to explain: you have to know about as much music as possible to make good recommendations. If you don’t know about an up and coming artist, you can’t recommend them. If you only analyze or rate or understand the popular stuff you by default fail at discovery. Manual discovery approaches by their nature do not scale. We track over two million artists and over 30 million songs and there is no way a manually curated database can reach that level of knowledge. Even websites that can be volunteer or community edited run against the limits of the community that takes part – we count only a little over 130,000 artist pages on Wikipedia. Pandora recently crossed the 1 million song barrier, and it took them 10 years to get there. Try any hot new artist in Pandora and you’ll get the dreaded:

Pandora not knowing about YUS

This is Pandora showing its lack of scale. They won’t have any information for YUS for some time and may never unless the artist sells well. This is bad news and should make you angry: why would you let a third party act as a filter on top of your very personal experiences with music? Why would you ever use something that “hid” things from you?

Activity data approaches (such as Last.fm and Amazon and iTunes Genius) also suffer from a slighter scale problem that manifests itself in a different way. It’s trivial to load a database of music into an activity data-based discovery engine (such as collaborative filtering or social tags.) I’ve often gone after such naive approaches to music discovery publicly. If a website or store has a list of user data (user A bought / listened to song Y at time Z) any bright engineer will immediately go into optimization mode. There’s almost a duplicitous ease of recommending music to people poorly. I recently was shopping for a specific type of transistor for a project on a parts and components website and found they, too, had turned on the SQL join that allowed “recommendations” on their site based on activity data:

Pathological filtering

Other than activity not making much sense in a discovery context, by default these systems suffer a “popularity bias,” where a lot of music simply doesn’t have enough activity data yet collected to be considered a recommendation match. Activity based systems can only know what people have told them explicitly, and this often makes it hard for less-popular artists to be recommended.

Care is a trickier concept and one we’ve tried very hard to define and encode into our engineering and product. I translate it as “is this useful for the musician or listener?”A great litmus test for care in music discovery is to check the similar artists or songs to The Beatles. Is it just the members of the Beatles and their side projects? For almost all services that use musical activity data, it will be:

Top artist similars are all members of the Beatles

Certainly a statistically correct result[3], but not a musically informative one. There is so much that user data can tell you about listening habits, but blindly using it to inform discovery belies a lack of care about the final result. Care is neatly handled by using social, manual or editorial approaches, as humans are pretty good at treating music properly. But when using more statistical or signal processing approaches that know about more music at scale, care has to be factored in somehow. Most purely signal processing approaches (such as Mufin here) fall down as badly on care as activity data approaches do:

Mufin expressing so little care about Stairway to Heaven

Care is a layer of quality assurance, editing and sanity checks, real-world usage and analysis and, well, care, on top of any systematic results. You have to be able to stand by your results and fix them if they aren’t useful to either musicians or listeners. Your WTF count has to be as low as possible. We’ve spent a lot of time embedding care into our process and while we’re always still working, we’re generally pleased with how our results look.

Echo Nest’s Beatles similars via MTV Music Meter

Without both care and scale you’ve got a system that a listener can’t trust and that musician can’t use to find new fans. You’ve failed both of your intended audiences and you might as well not try at all.

Care & Scale of common approaches

Text Analysis

Echo Nest Cultural vectors

I started doing music analysis work in 1999 at the NEC Research Institute in Princeton, NJ (I had scammed them into an internship and then a full time job by being very persistent.) NEC was then full of the top tier of data mining, text retrieval, machine learning and natural language processing (NLP[4]) scientists; I had the great fortune to work with guys like Steve Lawrence, Gary Flake, David Waltz and even Vladimir Vapnik moved into my tiny office after I left for MIT.

I was there while figuring out what to do with myself after abruptly quitting my PhD program in NLP at Columbia. I was a musician at the time, playing a lot of shows at various warehouse spaces or the lamented late “Brownies,” places where 20 people might show up and 10 would know who you were. There was a lot of excitement about “the future of music” – far more than there is today, as somehow we felt that the right forces would win and quickly. I logged onto Napster for the first time from a DSL connection and practically squealed in delight as a song could be downloaded faster than the time it would take to listen to it. It was a turning point for music access, but probably a step back for music discovery. We were still stuck with this:

Napster in 2001

The search was abysmal: a substring match on ID3v1 tags (32 characters each for artist, title, release and a single byte for genre) or filename (usually “C:\MUSIC\MYAWES~1\RAPSONG.MP3”) and there was no discovery beyond clicking on other users’ names and seeing what they had on their hard drives. I would make my music available but of course, no one would ever download it because there was no way for them to find it. A fellow musician friend quickly took to falsely renaming his songs as “remixes” by better known versions of himself: “ARTIST - TITLE - APHEX TWIN MIX” and reported immediate success.

At the time I was a member of various music mailing lists, USENET groups and frequent visitor of a new thing called “weblogs” and music news and review sites. I would read these voraciously and try to find stuff based on what people were talking about. To me, while listening to music is intensely private (almost always with headphones alone somewhere), the discovery of it is necessarily social. I figured there must be a way to take advantage of all of this conversation, all the excited people talking about music in the hopes that others can share in their discovery – and automate the process. Could a computer ‘read’ all that was going on across the internet? If just one person wrote about my music on some obscure corner of the web, then the system could know that too.

This is scale with care: real people feeding information into a large automated system from all different sources, without having to fill out a survey or edit a wiki page or join a social network.

After almost ten years of data mining, language and music research (first at NECI, then a PhD at MIT at the Media Lab) The Echo Nest currently is the only music understanding service that takes this approach. And it works. We crawl the web constantly, scanning over 10 million music related pages a day. We throw away spam and non-music related content through filtering, we try to quickly find artist names in large amounts of text and parse the language around the name. Every word anyone utters on the internet about music goes through our systems that look for descriptive terms, noun phrases and other text and those terms bucket up into what we call “cultural vectors” or “top terms.” Each artist and song has thousands of daily changing top terms. Each term has a weight associated, which tells us how important the description is (roughly, the probability that someone will describe music as that term.) We don’t use a fixed dictionary for this, we are able to understand new music terms as quickly as they are uttered, and our system works in many Latin-derived languages across many cultures.

On top of this statistical NLP, we also pull in structured data from a number of partners and community access sites like Wikipedia or Musicbrainz. We apply the same frequency and vector approach to this knowledge-base style data: if Wikipedia lists the location of an artist as NYC and the label partner as New York, NY and their Facebook page has “EVERYWHERE ON TOUR 2012”, we have to figure out which is the right answer to index. Often the cultural vectors on structured data become a synthesis of all the different data sources.

When a query for a similar artist or a playlist comes into our system, we take the source artist or song, grab its cultural vectors, and use those in real time to find the closest match. This is not easy to do at scale, and over the years we’ve done quite a lot of “big data” work to make this tractable. We don’t cache this data because it changes so often – the global conversation around music is very finicky and artists make overnight changes to their sound.

A lot of useful data naturally falls out of cultural analysis of music: the quantity of conversation is used to inform our “hotttnesss” and familiarity data points, representing how popular the artist is now on the internet and overall how well known they might be. We can use the crawled text anonymously as sort of a proxy for listener data without having to get it from a playback service. And the index of documents that relate to artists or songs is of course valuable to a lot of our customers in a feed or search context – showing news or reviews about artists their users are interested in.

Acoustic Analysis

Echo Nest acoustic analysis view

The internet is not the Library of Babel we envision it to be, and quite often many “lower-rank” (less popular) musicians are left out of the “cultural universe” we crawl. Also, the description of music necessarily leaves out things that actually describe the music – a Google blog search for Rihanna illustrates the problem well: many popular artists’ descriptions are skewed towards the celebrity angle and while this is certainly a valid thing to know about a musician, it’s not all we need to know. Lastly, internet discussion of music tends to concentrate on artists, not songs (although there is sometimes talk of individual songs on music blogs.) These three issues (and common sense) require us figure out if we can understand how a song sounds as well as how the artist and song is represented by listeners. And if we are going to follow care and scale, we’ve got to do this automatically, with a computer doing the job of the careful listening.

Can a computer really listen to music? A lot of people have promised it can over the years, but I’ve (personally) never heard a fully automated recommendation based purely on acoustic analysis that made any sense – and I’ve heard them all, from academic papers to startups to our own technology to big-company efforts. And that has a lot to do with the expectations of the listener. There are certain things computers are very good and fast at doing with music, like determining the tempo or key, or how loud it is. Then there are harder things that will get better as the science evolves, like time signature detection, beat tracking over time, transcription of a dominant melody, and instrument recognition. But even if a computer were to predict all of these features accurately, does that information really translate into a good recommendation? Usually not – and we’ve shown over the years that people’s expectation of “similar” – either in a playlist or a list of artists or songs – trends heavily towards the cultural side, something that no computer can get at simply by analyzing a signal.

But it does turn out that acoustic analysis has a huge part to play in our algorithms. People expect playlists to be smooth and not jump around too much. Quiet songs should not be followed with loud metal benders (unless the listener asked for that.) For jogging, the tempo should steadily increase. Most coherent mixes should keep the instrumentation generally stable. Songs should flow into one another like a DJ would program them, keeping tempo or key consistent. And there’s a ton we haven’t figured out yet on the interface side. Could a “super dorky query interface” work for music recommendation, where a listener can filter by dominant key or loudness dynamics? Maybe with the right user experience. An early product out of the Echo Nest[5] was an “intelligent pause button” that Tristan whipped up that would compose a repeating segment out of the part of the song you were in or just play the song roughly forever (check out an automated 10 minute MP3 re-edit of a Phoenix song) – which a few years later became Paul’s amazing Infinite Jukebox – these experiments are fascinating precursors to a new listening experience that might become more important than discovery itself.

How Echo Nest acoustic analysis works

The Echo Nest audio analysis engine (PDF) contains a suite of machine listening processes that can take any audio file and outputs both low-level (such as the time of when every beat starts) and high-level (such as the overall “danceability”) information for any song in the world. We analyze all the music we work with, and developers can upload their own audio to see everything we compute on a track via our API. Our analysis starts by pretending it was an ear: it will model the frequencies and loudness of a musical signal much the same way perceptual codecs like MP3 or AAC do. It then segments the audio into small pieces – roughly 200ms to 4s, depending on how fast things are happening in the song. For each segment we can tell you the pitch (in a 12-dimensional vector called chroma), the loudness (in an ADSR-style envelope) and the timbre, which is another 12 dimensional vector that represents the sound of the sound – what instruments there are, how noisy it is, etc. It also tracks beats across the signal, in subdivisions of the musical meter called tatums, and then per beat and bar, alongside larger song-level structure we call sections that denote choruses, intros, bridges and verses.

That low level information can be combined through some useful applications of machine learning that Tristan and has team have built over the years to “understand” the song at a higher level. We emit song attributes such as danceability, energy, key, liveness, and speechiness, which aim to represent the aboutness of the song in single floating point scalars. These attributes are either heuristically or statistically observed from large testbeds: we work with musicians to label large swaths of ground truth audio against which to test and evaluate our models. Our audio analysis can be seen as an automated lead sheet or a computationally understandable overview of the song: how fast it is, how loud it gets, what instruments are in it. The data within the analysis is so fine grained that you can use it as a remix tool – it can chop up songs by individual segments or beats and rearrange them without anyone noticing.

We don’t use either type of data alone to do recommendations. We always filter the world of music through the cultural approaches I showed above and then use the acoustic information to order or sort the results by song. A great test of a music recommender is to see how it deals with heavy metal ballads – you normally would expect other ballads by heavy metal bands. This requires a combination of the acoustic and cultural analysis working in concert. The acoustic information is obviously also useful for playlist generation and ordering, or keeping the mood of a recommendation list coherent.

What’s next

I’ve used every single automated music recommendation platform, technology or service. It’s obviously part of my job and it’s been astounding to watch the field (both academically and commercially) mature and test new approaches. We’ve come a long way from RINGO and while the Echo Nest-style system is undoubtedly the top of the pack these days as far as raw quality of automated results go, there’s still quite a lot of room to grow. I’ve been noticing two trends in the space that will certainly heat up in the years to come:

Social – filtering collaborative filtering

This is my jam

Social music discovery as embodied by some of my favorite music services such as This is my jam, Swarm.fm, Facebook’s music activity ticker and real time broadcasting services like the old Listening Room and its modern progentior Turntable.fm often have no or little automated music discovery. Friend-to-friend music recommendations enabled by social networks are extremely valuable in music discovery (and I personally rely on them quite often), but are not recommendation engines as they can not automatically predict anything when bereft of explicit social signals (and so they fail on scale.) The main amazing and useful feature of recommendation systems are that it can find things for you that you wouldn’t have otherwise come across.

Even though that puts this kind of service outside the scope of this article, that doesn’t mean we should ignore the power of social recommendations. There’s something very obvious in the social fabric of these services that makes personal recommendations more valuable: people don’t like computers telling them what to do.

There are some interesting new services that mix both the social aspects of recommendation with automated measures. This is my jam’s “related jams” is a first useful crack at this, as is Spotify’s recent “Discover” feature. You can almost see these as extensions of your social graph: if your friends haven’t yet caught onto Frank Ocean there might be signals that show they will get there soon, and using cultural filtering can get us there. And a lot of the power of social recommendations – that it comes from your friends – can tell the story better than just a raw list of “artists we think you would like.”

Listener intelligence

When do you listen to music? Is it in the morning on your way to work? Is it on the weekends, relaxing at home? When you do it, how often do you listen to albums versus individual songs in a playlist? Do you idly turn on an automated radio station, or have your own playlists? Which services do you use and why? If it’s raining, do you find yourself putting on different music? The scariest thing about all the music recommendation systems I’ve gone over (including ours as of right now) is none of them look at this necessary listener context.

There is a lot going on in this space, both internal stuff we can’t announce yet at Echo Nest, and new products I’m seeing coming out of Spotify and Facebook. We’re throwing our weight behind the taste profile – the API that represents musical activity on our servers. It’s sort of the scrobble 2.0: both representing playback activity as well as all the necessary context around it: your behaviors and patterns, your collection, your usage across services and maybe even domains. We’re even publishing APIs to do bulk analysis of the activity to surface attributes like “mainstreamness” or “taste-freeze,” the average active year of your favorite artists. This is more than activity mining as collaborative filtering sees it: it’s understanding everything about the listener we can, well beyond just making a prediction of taste based on purchase or streaming activity. All of these attributes and analysis might be part of the final frontier of music recommendation: understanding enough to really understand the music and the listener it’s directed to.

– Brian (brian@echonest.com)

Thanks to EVB for the edits

I know they do a ton of acoustic analysis there but I don’t know if it’s used in radio or similar artists/songs. Probably. ↩
They use Gracenote’s CDDB data as well but I don’t know to what extent it appears in Genius. ↩
You can even claim that since you are listening to the Beatles, you are also listening to the individual members at the same time but let’s not get too ahead of the curve here ↩
Whenever I say NLP I mean the real NLP, the natural language processing NLP, not the creepy pseudoscience one. ↩
Released eventually as the Earworm example in Echo Nest Remix. ↩

Music data talk at Velocity EU

Tue, 23 Oct 2012 11:39:00 -0400

I gave a talk at Velocity EU in London a couple of weeks ago on how some pieces of the Echo Nest work, operationally:

(Direct link on speakerdeck)

We started Echo Nest seven years ago and I couldn’t imagine a stranger time to begin a technology-focused venture. We started with 2 self-built 2U rack machines in a closet at MIT and within two years started moving everything to first dozens and then almost thousands of virtualized hosts running on a few different cloud providers. And then a year or two ago we pulled it all back to physical again. We’re in a weird spot between offline data processor and real time API provider and none of the oft-repeated hype platforms ever worked for us.

We’ve been around before and during much of Hadoop, Solr, “NoSQL”, EC2, API-as-PR, the rise of mobile, Apple on Intel, Hacker News and sharded mySQL as a key-value store. And we’re still at it, making money in an industry that would rather keep it to itself. If it’s not obvious, I’m really proud of what we built and the small team that works their ass off to keep it working while making some really cool new stuff.

There’s a lot of stories to tell and this is just the first one, on how we’ve abused text indexing to do quite a lot of our work. If you ever wondered how some of the musical data souffle avoids burning, here it is. Thanks to John & Kellan for giving me a chance to present this.

1980s pen plotters of the future

Sun, 12 Aug 2012 09:38:34 -0400

Early 1980s pen plotters are amazing tools that are still very useful in today’s world. There’s something completely transfixing about a mechanical device moving an actual pen on paper versus the smelly black box of the laser printer. And if you’re trying to draw lines or curves or like the effect of actual ink touching paper (not sprayed on in microdots) there’s no other way. Luckily, there’s some great tools out there for making plotters work on modern hardware and using modern file formats (PDFs) and the hardware itself, while finicky and aging, is cheap.

Hardware

You’re going to have to start with the right hardware. The Chiplotle! plotter FAQ is great for this, I’ve added my notes below:

USB - serial interface or a serial port on your computer. I have trouble with Chiplotle! with the very common Prolific serial adapters (it might be the drivers) but if you have something that works it’ll probably be fine. I use an ex-Keyspan one.
Any HPGL compatible pen plotter. eBay is almost always your best bet, unless you live near the MIT Flea Market. Make sure the plotter supports HPGL via serial connection. If it says HPIB or GPIB, do not get. Very common eBay finds are the Roland DXYs or the HP 7475a. If in doubt, check the Chiplotle! list of supported plotters (although keep in mind similar model #s will also work; for example the DXY-1150 and 1150a act as a DXY1300 in Chiplotle!.) I normally pay $50 or so for a single sheet plotter like the DXY-1150. Make sure you get a power supply with the plotter, that’s often the hardest thing to find and none of them use anything standard.
A plotter serial cable. The only place you’ll need to pull out a soldering iron. You can buy plotter serial cables on eBay but it’s easier to just make your own from a DB25 male to DB9 female cable. You have to re-route a few wires but it’s easy to do.
Pens & paper. Your eBay plotter likely will come with a plastic bag full of dried out pens or, if you’re lucky, boxed ones. There’s a huge variety of pen types (for different paper or thicknesses, felt vs. fine point) or you can fashion your own if you’re wily. For paper, the average desktop plotter can take up to 11 x 17” paper or just normal printer paper (make sure to set the DIP switches on the back if you don’t use the full size.) I have a nice stack of artist vellum paper with nice vellum pens.

Have all that? Now, find a place to put the plotter, run the plotter cable from the serial port on back of the plotter to your USB adapter, and load up some paper. Depending on the plotter, the paper might be held in electrostatically or it may be magnetized and expect a metal tab to hold in the paper (which you obviously did not get in the eBay shipment, you can use a metal ruler or something similar instead.) Or maybe you will have to tape the paper on. If it’s a wide format plotter the paper will have to be in a roll and the vertical axis is done via the roller mechanism like a receipt printer (these are notriously fiddly, I would avoid them unless you really need 3 foot wide paper.) You then load up a pen (in the pen holder off to the side, not the plotter head — the beginning of every HPGL file tells the plotter to pick up the pen from the holder.) Now, let’s plot.

Software

In 2008 I was bitten by the plotter bug all of a sudden. I was trying to draw a smooth bezier curve robotically and was looking at various servo or motor solutions when I stumbled on the community of folks that have adapted Roland pen plotters into vinyl cutting CNC machines. I found myself intensely bidding on my first plotter against a familiar eBay username. After I lost, I confirmed my suspicions: I was in competition with my dear friend Douglas Repetto of CMC & dorkbot fame. And not only was he also independently plotter crazed, he was working on a Python module for HPGL control called, well, Chiplotle!. Maybe there was something in the water that week.

Chiplotle is obviously the best and only way to reliably control a plotter from a modern computer. It does quite a lot of work for you: it manages the commandset of your plotter, buffers output so it doesn’t overflow and start drawing random straight lines, provides a interactive terminal where you can “live draw” and a bunch of other necessary stuff. Although you can import chiplotle into your program and programatically control your plotter, I tend to use just one commandset of Chiplotle — the plot_hpgl_file script that it installs.

HPGL files are like the wizened ancestor of PDF. It is simply an ASCII file of text commands to draw lines, curves, choose pens, etc and is the plotter’s native language. If you want, you can ignore Chiplotle! altogether and just cat an HPGL file to your serial port at 9600 baud, 8N1. This will work fine for the first few commands but eventually the plotter’s internal buffer (mine is 512 bytes) will overflow. plot_hpgl_file takes care of all of this. The first time you run it, it will attempt to detect which plotter you have on which serial port. Then it will slowly spit out the HPGL commands and make sure the plotter is acknowledging them.

My workflow for the project I am on now is to generate PDF files programatically using the amazing ReportLab python PDF toolkit and then convert them to HPGL using pstoedit and plot it. It is as simple as

python your_pdf_generator.py file.pdf
pstoedit -f hpgl file.pdf > output.hpgl
plot_hpgl_file.py output.hpgl

Obviously I could have my python program directly control the plotter, but as ink and paper add up, you will want a step in between to make sure your art is OK, and the PDF step is natively viewable on any platform. Since PDF and HPGL both share a lot of common ancestry, the curveTos, lineTos and moveTos are kept consistent with no loss of quality. There’s no rasterizing step: if you generate curves programatically with ReportLab, they will be the same curve on the paper in the plotter.

Musical introversion - an ode to This is My Jam

Thu, 09 Aug 2012 10:44:28 -0400

Without a doubt my favorite internet music service ever is This is my jam (aka TIMJ), a self styled “slow music” site where you get to choose one song only for your friends to hear. The song expires in seven days if you don’t change it. That’s it.

It’s been out for about six months and it’s doing great; these guys do not need my publicity help and I doubt anyone reading this on my blog needs to hear about it again. And all four that built it are dear friends & there is a tenuous relationship between my company and theirs.¹ You may think there’s a lot of bias here. But I am not the kind of person to blindly like something because I was involved, in fact, for me it usually means they have to work even harder to get over my inner critic. And even worse for them, I tend to hate almost every music social network. I grew up with music as the most personal and private thing in my life. I normally refuse to answer the “what music do you like” question, and Spotify’s recent “social listening” features really put me off the service. I also (naturally) hate most attempts of a computer trying to tell me what to listen to.

But TIMJ has somehow won a place in my heart. I “follow” three times more people there than Twitter, and I spend roughly two hours on the site a day, listening to my friend’s jams and sometimes updating mine. What TIMJ does so well I think comes down to three things:

They make you look good

This is My Jam looks great, and the hallmark of a great social experience is that it also makes you look good without much effort. The jam selection is very visual, almost forcing the choice of a “cover image” (square) and background (widescreen.)

I doubt I am not the only TIMJ member with this folder on their desktop:

Hannah (TIMJ’s lead designer) deserves a huge shout-out for her amazing work on the site. I happened to be in London with them the day they began work on the concept in earnest, and the first thing she did was go in another room (where I happened to be peeling a mango) and draw three pages full of circles. I’m a computer person forever, but I know when someone has put care and serious thought into something and you easily get the feeling that every pixel and click has been thought through.

They’ve been the first to really do “frictionless” right

For many years myself and my remote brain The Echo Nest have been harping hard on the music platform — the idea that the provider of audio can be decoupled from the music experience itself. This was a pipe dream only a few years ago, but now we’ve got Spotify apps, Soundcloud embeds and Rdio APIs. And TIMJ is truly the first site I’ve seen that makes real use of these. When you want to create a new jam, you don’t need to futz with local files on your computer or figuring out what service to use. The jam search seamlessly shows you two lists: one of just audio (you provide your own image) or one of video with audio. Obviously, backing that is APIs from Soundcloud, YouTube, Hype Machine, and Echo Nest but no one needs to know or care about that.

And of course, as of today, TIMJ has a new Spotify app:

I’ve been playing with the Spotify app for a while because I had access to their GitHub repository² and the playlists it creates of your likes (and friend’s jams & likes) have become my main source of new music on Spotify. Team Jam made a great decision to use Spotify as the archive read-only mode of TIMJ, the place where you go when now is too much to deal with. In practice, Spotify seems to match a large majority of jams already in its database.

Slow and steady wins the race

There is nothing to stop you or your friends from posting new jams every three minutes, but there are quiet hints throughout the site to keep it slow, mostly ambient clues like the lack of a “stream” auto-updating view and the relatively many-click process of updating your jam, or the fact that you can’t easily do it from mobile or SMS or other networks. I don’t know the actual numbers but I’m guessing the average “jam length” (the amount of time a users’ jam stays active) is about 2-4 days. This maps well to my musical brain. I can’t stand Spotify/Facebook’s incessant scroller of activity, music deserves a lot more care than that. And hinting the speed makes your friends care about what they post and only surfaces the best stuff.

Another great (surely frustrating at first) hint is the focus on “now” versus the always tempting archival nostalgia past. Only the current jam per person is listen-able on the site, and only the past 5 are even shown. All comments, likes and plays disappear when a new jam shows up. It’s the opposite instinct that every other social experience tries to have and it works great for (the very temporal and fiddly) music.

Congrats to Team Jam on the Spotify app release and for the stunning growth and for making my favorite music site ever. And make sure to follow me on TIMJ, I have the best taste in music ever.

Simply put, Echo Nest currently ‘sponsors’ it but it’s an independent entity; in exchange for the domain name I bought in 2007 for something totally different, they have to let me sit in their office in London sometime. ↩
At some point it was rewritten to require some sort of Makefile written in Brunch and every single node.js module ever — for what amounts to a web page — I truly am no longer made for these times ↩

Green & blue pixels from CRTs

Sun, 05 Aug 2012 18:45:00 -0400

Green & blue pixels from CRTs

Owl study one, two three

Sun, 05 Aug 2012 12:28:20 -0400

Owl study one, two three

I spent a good six hours building this Apple //c Information...

Wed, 25 Jul 2012 16:30:46 -0400

I spent a good six hours building this Apple //c Information Kiosk. It’s hilarious! You can type in any band name (that existed 1984 or earlier OF COURSE) and it shows you a picture and their bio. Or in automatic mode just scrolls through a set of pop artists around that time (via this amazing Echo Nest API call.

You can run it yourself if you have the right hardware, here’s my HOWTO.

Today in the fabulous uses of audio technologies department, the...

Mon, 23 Jul 2012 22:28:16 -0400

Today in the fabulous uses of audio technologies department, the bkstreetmusic Twitter account

The audio fingerprinting at the Echo Nest FAQ

Sun, 22 Jul 2012 20:24:00 -0400

One of the most popular APIs at the Echo Nest is for our two open audio fingerprinting systems — ENMFP or Echoprint. Between them both, we currently process queries for over 40 million songs a week— almost 70 fingerprints a second. We announced ENMFP two years ago and Echoprint one year later and it’s been going very fast since then. The world seemed to need an open music identification service and we were happy to provide it.

I co-developed both FP technologies (ENMFP using Tristan’s Echo Nest Analyze and Echoprint with Dan Ellis at Columbia) and often get the collected questions from our developers, customers and interested parties. There’s a lot of good questions that we haven’t answered very well anywhere else on our website or developer docs, so I thought I would collect them all in one place in sort of a living document on my blog.

Echo Nest fingerprinting FAQs

Last updated: July 22 2012

Some definitions to help
What fingerprinting technologies does the Echo Nest provide?
How much do they cost? Can I use them anywhere? Commercially? What about the data or the server
Can they fix my (or my users’) metadata on files on a hard drive?
I have a ton of audio (I’m a streaming service or something). Can I just run my own Echoprint server and codegen and compute my own codes and never talk to you again?
How much of the original audio do I need to match? Do I have to start at the beginning?
Once I have audio, how fast can I identify a file?
Can I make it work like Shazam or SoundHound?
Echoprint is newer. Will ENMFP still be supported?
Are there any whitepapers or more information on how they work?
Can I run ENMFP or Echoprint on a mobile device?
I tried song/identify and got no results!!
What type of audio does fingerprinting work on?
Can audio fingerprinting be used to detect cover versions or live versions of songs? What determines the same song?
Where can I go for help?

Some definitions to help:

code generator (or often “codegen”): a piece of software that runs on a computer, mobile device or server that computes “codes” from an audio stream meant for sending to a fingerprint lookup server for a possible match.
query server: a stack that receives codes and searches through data to find a match
matching data: the set of resolvable song codes that a query server can match, given query codes. Matching data must be computed on the whole song as query codes can be from anywhere in the song.

What fingerprinting technologies does the Echo Nest provide?

We have two: ENMFP and Echoprint.

How much do they cost? Can I use them anywhere? Commercially? What about the data or the server?

ENMFP:

Code generator is free to use for anything — commercial or otherwise. The codegen binary for Windows, Mac OS X and Linux (there is no source available) is available here, although you must have an active Echo Nest Developer account.
Server code is not available¹. The Echo Nest maintains the only ENMFP server via the song/identify API with a version number of 3.15.
The matching data has roughly 35 million unique songs and is not available publicly.
The song/identify call can be used by anyone, even commercially (there is an exception in our terms for this) but you will be under a rate limit of around 120 calls per minute, subject to change.

Echoprint:

Code generator is free to use and open source.
Server code is also free to use and open source.. The Echo Nest maintains an Echoprint server via the song/identify API with any version number set to 4.0 or higher.
The matching data is available for download with the intent that outside sources can add to it later. The data is small at the moment — roughly 200,000 songs, but will be growing very soon.
The song/identify call can be used by anyone, even commercially (there is an exception in our terms for this) but you will be under a rate limit of around 120 calls per minute, subject to change.

Can they fix my (or my users’) metadata on files on a hard drive?

For ENMFP, absolutely. You have a matching database of most music available for you. You can quickly test this out on your computer’s music. First, download the ENMFP codegen and run this:

Then, run this python script or something like it to look up each code:

(Obviously, you’d want to thread this call out in real world use.) There, you’ve built a scan and match service! Sell it for $50 each on the Mac App store.

For Echoprint, yes, although the matching database is still small.

I have a ton of audio (I’m a streaming service or something). Can I just run my own Echoprint server and codegen and compute my own codes and never talk to you again?

Yes. In fact, a lot of very big companies are doing this.

How much of the original audio do I need to match? Do I have to start at the beginning?

For ENMFP, we suggest 50 codes worth of audio, which is usually between 15 and 20 seconds. For Echoprint, we suggest 20 seconds worth of audio. The audio can be from anywhere within the song.

Once I have audio, how fast can I identify a file?

ENMFP’s codegen computes in roughly 20x real time — for a 30s sample it will take 1.5 seconds, for example. Echoprint’s codegen is roughly 1000x real time.

The server side varies on load and database side, but we aim to keep both within a 500ms response and support 50 queries a second per server. Those booting their own servers can easily shard or mirror multiple servers in a load balancer.

Can I make it work like Shazam or SoundHound?

This usually means “will over-the-air queries work,” as in a bar or noisy place. For ENMFP, no. ENMFP is only supported to work on files or “clean audio.” For Echoprint, this is the intent, although as of right now we do not support that.

Echoprint is newer. Will ENMFP still be supported?

Yes. The only thing that may change is the name, as it’s getting annoying to talk about two separate fingerprints. Even once we have a real catalog backing Echoprint, ENMFP still has some good properties that are useful for a lot of our customers.

Are there any whitepapers or more information on how they work?

Very short ones: here’s Echoprint’s and here’s ENMFP’s.

Can I run ENMFP or Echoprint on a mobile device?

There are two ways to run a fingerprint on mobile: one is to compute the codes on the device and query the server, another is to send a low bandwidth audio stream to another server, which computes the codes and then sends it to the server. Both ENMFP and Echoprint can run in the latter.

ENMFP is too compute-intensive to run the codegen on the device and we do not provide a codegen that runs on any mobile.

Echoprint can easily compute codes on a device, and we provide an example Xcode project for iOS.

I tried song/identify and got no results!!

The main problems are almost always one of the following:

You have Echoprint codes but are not specifying the version # (has to be 4.0 or over) in the call
You have ENMFP codes but are specifying a >4.0 version number.
You are running the example codegen for either and your version of FFMpeg is not set up to decode MP3 files.
You are using Echoprint and the match is not in our test 200,000 song database.

What type of audio does fingerprinting work on?

ENMFP and Echoprint were originally intended to provide fingerprinting for musical audio, and that is primarily what they are used for (for some definition of “music”). However, several of our users have also reported success stories about using Echoprint on speech audio.

Can audio fingerprinting be used to detect cover versions or live versions of songs? What determines the same song?

No. ENMFP and Echoprint are designed to identify the “same” rendition or recording of a particular song. For the purposes of audio fingerprinting, cover versions and live versions are considered to be “different”, and hence queries will be identified as such.

A song is defined by being the same recording or having the same master recording. Both fingerprints will consider a remastered version the same song, will consider a radio edit the same song, and will consider different encodings (e.g. MP3 at different bitrates) the same song.

Where can I go for help?

If you are using either FP system, best to:

Join the Echoprint google group. Best place to discuss either.
Watch or post to the Echo Nest Developer Forums if you have any trouble with the API (not the codegen or server code.)

The server stack between ENMFP and Echoprint is very similar and it’s very possible a dedicated developer can make one work like the other. But not supported. ↩

How well does music predict your politics?

Thu, 12 Jul 2012 10:55:13 -0400

Mitt Romney with Kid Rock

Maybe you should just plug your iPod into the booth or connect the Diebold machine to Facebook this November. It started as an office joke, but after running the numbers, I can’t escape the data. It turns out music preference is pretty well correlated with political affiliation.

A few highlights:

Republicans seem to have less diverse music taste than Democrats
Kenny Chesney fans are most likely to swing right, Rihanna fans left.
Metal fans could save us all from a two-party system.

Taste Profiles

Over the past few years, we here at The Echo Nest have put a lot of engineer energy into “Taste Profiles” – our server-side representation of everything a listener does with music. We first released them in 2010, then we linked them to our playlist API with catalog-radio, and recently released a large amount for research. Anyone can store their musical activity across services in our “cloud” from metadata, audio files or fingerprints, and update stats like play counts, skips, ratings, loves and bands. The Taste Profile ID can then do all sorts of stuff – everything from recommending you new music, to syncing your local collection to a cloud service, to suggesting shows to see, to some of the crazy stuff I’ll walk through below.

Quite a lot of our API developers and customers are already using Taste Profiles to manage the musical identity of their customers, and we use them internally as the proving ground for a lot of our analytics work. For example, we spent the last few months making sure people listen to our automated radio service (used in iHeartRadio, Spotify, VEVO, MOG and many others) as long as possible by predicting how they’ll respond to our suggestions. And with today’s release of new Taste Profile key-values you can now annotate a Taste Profile with any information you want (such as your location, the device you’re using, your IDs on social networks or anything else) and we’ll use it to give you better results. Part of the push behind arbitrary data in a musical identify is to track how “non-musical behaviors” can make our results better. If you live in Sweden, maybe you don’t want to hear ABBA anymore. Or you’ve just seen a Wes Anderson film — we might want to send you down a Kinks path.

We’ve been collecting this (completely anonymized of course) data for a while now and started looking into what correlations exist between music, psychographics, demographics and other media preferences. As the time is upon us in the US to start thinking about who to elect as president, we thought we’d first look to see what political affiliation data we had and if it had any correlation to music. Can we tell if someone is a Republican just from his or her iTunes collection? And if so, which artists are the key “tells” for both sides? What we found was fascinating.

Predicting politics from Taste Profiles

Barack Obama with BB King

Some couching notes before we get started: although we have far more data, we’ll only look at listeners that self-report as either “Democrat-aligned” or “Republican-aligned” and are living in the US, to make it a quicker read.[1] The political alignment was automatically derived from annotations of political figures or parties, we grouped prominent political figures such as Bill or Hillary Clinton, Barack Obama, John Kerry and so on as Democrat alignment, as would be a specific party affiliation. Likewise for republican: George Bush, Mitt Romney, John McCain, Sarah Palin, and so on. If someone listed both or had conflicting affiliations we did not include them in this experiment, nor did we consider any other party or country.[2] Finally, throughout this I’ll call the two classes “Democrats” and “Republicans,” even though that’s an incredibly generalized version of the available data.

So let’s get to it: let’s take a bunch of Taste Profiles, see which ones have political affiliation listed and then try to learn the relationship between the musical data in the Taste Profile and the affiliation.

What kind of musical data are we talking about here? An Echo Nest Taste Profile can be as simple as a list of artists the listener likes or can include very detailed information about his or her listening activity. The Echo Nest then of course has tens of thousands of points of data to associate with each artist or song. These data points are as mundane as the name of the artist (“Carly Rae Jepsen”) to as complex as the number of millseconds in between each downbeat (4508), or the predicted key (E major), or the probabilities of words people use to describe the artist on the internet (“angular,” “stupid,” “witch house.”)[3] We use all of this data to recommend you music on MTV.com or play you a great station on iHeartRadio, and here we’re going to use it to see if you like big government.

Cultural vectors used in Echo Nest musical analysis, here for “ABBA”

For every person’s Taste Profile, we have many thousands of terms that describe the kind of music the person is into. For this experiment, half of the Taste Profiles’ worth of musical term data is thrown into The Echo Nest’s statistical machine learning classifiers (we have our own custom stuff that mostly acts as a very large scale multi-class support vector machine.[4]) and associated with the “ground truth” of affiliation to try to learn a model of each class. In layman’s terms, this basically means we show the system a bunch of examples of Democrat Taste Profiles and a bunch of Republican ones and see if it can predict the class on a new, unknown Taste Profile. Our machine learning tech is good at handling messy data like this— we’ve added a lot of math and magic on top to deal with our specific kind of musical data.[5]

The results

After we’ve learned the model we can then test it by giving it the other half of the data — asking our classifiers to identify each previously unknown Taste Profile as each Democrat or Republican — to see how well it does at prediction. We use a few measures to evaluate the experiment:

raw accuracy (out of all the test examples, how many did we get right)
precision (out of the ones we predicted in the class, how many were in that class)
recall (out of all the examples of that class, how many did the classifier find) and
F1, a blended measure of precision and recall that people in this field like to use as a general performance metric.

Prediction accuracies

Looks like we’ve got something here! We hit an F1 of over 0.8 for Republican prediction and just under 0.4 for Democrat prediction. These are both good numbers, in line or above many other prediction algorithms. But why is our Democrat prediction less than half as accurate as our Republican prediction? Shouldn’t they be the same?

Musical diversity

Political affiliation is not binary, and it’s not like we can assume that just because someone didn’t explicitly list an affiliation with the Democratic party that they are voting for Romney. And it turns out the correlation between musical preference and Democrat affiliation is slightly harder to tease out. When I was going over the data I had a theory: Republicans might listen to fewer kinds of music. If most of the class you are trying to predict stays within a narrow range of music types, they’re easier to spot. Conversely, if the class you’re predicting is all over the musical map, it becomes harder to make accurate predictions. And the data shows it. If we add up the occurrence of each musical term associated with each person (for example: Joe listens to “rock” at “110 bpm” that sounds like “Aerosmith” or “the 70s” and is voting for Romney, we mark a +1 under each of those terms for the Republican bucket) and then plot the histogram counts in descending order, we see a clear difference in both the magnitude and distribution of musical types for the two political affiliations.[6]

Histogram counts of the top occurring musical terms for each class.

Not only are there less musical types overall, but the clear almost right angle “elbow” of the Republican histogram distribution shows that after a small set of top ranking terms, their listenership tends to have far less musical diversity. The Democrat curve is smoother, indicating that those people listen to more types of music overall. Overall, for every 10 unique musical types Democrats listen to, Republicans listen to just 7.

As silly as it might seem to predict political affiliation with music, this is a great example of why we do this kind of analysis. Now that we’ve found room for improvement in predicting a class by looking at diversity as a feature, we can apply this in our work to make our playlists and recommendations better. We’re currently adding a “diversity index” to our Taste Profile back end to model this exact effect.

Who are Kenny Chesney fans voting for?

Another fun thing to look with these political classifiers is the inputs that the machine thought were the most predictive of political affiliation. For each class, you can do some quick math on the model by inspecting the margin. This tells us which musical terms and properties best separate each political class:

In this diagram of the support vector machine in action, the circles on the dotted lines are the support vectors where training examples are used to define the classification boundaries.

Since a list of probabilities of music vectors isn’t exactly good blog material, I found the closest matching artists to each set of terms to show here. We get a great list from doing this:

Artists whose fans are most correlated to Republican

1. Kenny Chesney
2. George Strait
3. Reba McEntire
4. Tim McGraw
5. Jason Aldean
6. Blake Shelton
7. Shania Twain
8. Kelly Clarkson
9. Pink Floyd[7]
10. Elvis Presley

Artists whose fans are most correlated to Democrat

1. Rihanna
2. Jay-Z
3. Madonna
4. Lady Gaga
5. Katy Perry
6. Snoop Dogg
7. Chris Brown
8. Usher
9. Eminem
10. Bob Marley

The Democrat high predictive list reminds me of another experiment I ran: taking the Billboard Top 10 and feeding it through the predictor. As of this writing the top artists are Carly Rae Repsen, Maroon 5, Gotye, Katy Perry, Rihanna, Ellie Goulding, fun., Nicki Minaj, David Guetta and Usher — almost all artists that skew very strongly Democratic. If only people that buy singles in music stores vote this November, it will be a complete Obama landslide.

Classifier confusion prediction

Lastly, I thought the “highest confusion” artist list was interesting. These artists are not good predictors of either Republican or Democrat, so if you like them, you’re relatively safe from this Minority Report shit.

Artists whose fans are hardest to predict for either Democrat or Republican

1. The Beatles
2. Marilyn Manson
3. The Rolling Stones
4. Johnny Cash
5. Pantera
6. Alice in Chains
7. Paradise Lost
8. Moonspell
9. Fleetwood Mac
10. Tiamat

I found it neat these non-predictive artists were mainly metal. Perhaps the genre that can finally bring this divided country together or break the lock on the two party system.

OK, now what?

Obviously, The Echo Nest is not going to quit our day jobs to become pollsters. But it’s fascinating how much information about you is sitting inside your musical tastes, and we also appreciate how we can use these experiments to tune our models to make your listening experience better on everything we power (Clear Channel’s iHeartRadio, eMusic, MOG, Spotify, Nokia, the BBC, VEVO, and many more.) We have a lot more ideas around this angle. Stay tuned to this blog for more experiments, and please let me know if you’ve got any great ideas.

Huge thanks to proofreaders / reviewers Dan Ellis, Eliot van Buskirk, Jim Lucchese

All data used in this experiment (and all Echo Nest Taste Profile data used in analytics) was carefully anonymized before we received it. ↩
Please get in touch if you’ve got a great idea for another experiment on data like this that I don’t cover! ↩
Perhaps fit for another blog post, but if you’re reading down here: it turns out that the musical data the Echo Nest has is crucial for the experiment to work. Just training a model against artist or song names alone did not work nearly as well. F1s dropped on average of 20–40% depending on the task. ↩
In practice we use a somewhat “reduced” form of the SVM known as RLSC so that we can easily scale across many machines, but the intent is the same. You can read about RLSC on music data in an old paper of mine if you’re interested. (PDF) ↩
We trained two binary classifiers: Rep vs. Dem and Dem vs. Rep. The training data chose a large random sample from the universe of data, which was 66% Dem and 34% Rep. ↩
To make this histogram chart, we normalized both bucket counts first by the universe of terms to ensure this bias in the priors would not affect the distribution. ↩
One day, Echo Nest will start delineating between “good” Pink Floyd (Syd) and the other kind ↩

Why music ID resolution matters to every music fan on Facebook

Tue, 11 Oct 2011 08:57:00 -0400

Facebook’s music announcement a couple of weeks ago was a huge land grab, an audacious move to get itself ensconced as the nexus of that music platform I’ve been talking about. On paper and on stage the service looked game changing: all your music players and services all in one place, neatly collected with your friends to help you navigate the massive world of music. Myself and my engineering team at The Echo Nest have probably spent as much time thinking about that massive world of music as anyone on earth, so I thought I’d try putting it through its paces.

Facebook’s recently-launched music service now shows every music fan why the crazy and complicated world of music ID resolution matters to all of us. The more social our music activity becomes, the more music data becomes relevant to music fans every day.

Though Facebook music has only been live for a couple of weeks, Facebook is clearly struggling with some well-worn challenges in music ID resolution — problems I’ve been dealing with for many years now. Below are some examples highlighting the promise of Facebook music and some common music ID resolution problems they’ll need to fix to really deliver on the promise…

The holy grail — ‘universal song ID’

The only way social music features can work is if the song you want to hear actually plays.

If you’re listening to a song in Spotify, it will broadcast to Facebook and all your friends see what you are listening to in real time⁰. If they want to play along, your friends click a simple play button to hear it themselves. No Spotify? No problem— Facebook launched with a huge array of music content partners (with some conspicuous elephants missing) and, if that song is available in your choice of player, it will play with no “friction.” You’ll see something nice like:

Which opens the song in your selected player (in this case, it’s Rdio, which just needs to be open in your browser.) This is inconceivably great for music consumers— there’s a lot of great competing services, all with unique features and cost structures, and giving more choice is always good. I may want to use MOG because it has excellent Echo Nest-powered discovery features. Or Rdio or Spotify because they allow 3rd party mobile apps. Or Slacker or iHeartRadio for a radio experience.

What song is this?

This is all music-world changing stuff, if it worked. When I first played with Facebook Music, I tried a simple example. I put on the most terrible popular song I could think of in my Spotify player:

And then I asked Facebook to “Play in Rdio.” I heard something sort of like it but not exactly:

Here Facebook has decided that Rdio’s version of “Your Body is a Wonderland” is a sounds-like version from “The Hit Crew.” I cannot think of a worse fate: hearing something worse than John Mayer when you have to click on a link that says John Mayer. (Consider clicking on a Google search result for your dentist’s office phone number and getting your ex-girlfriend instead.)

At The Echo Nest, we know “The Hit Crew” all too well; they crank out “soundslike” versions of tunes. This is a great example of a basic music resolving problem: every song in any reasonably sized catalog has dozens of karaoke versions, covers, instrumentals, yoga mixes, etc. For Facebook to resolve a top 20 single to its sounds-like version is pretty ugly. What’s going on?

Moving on, that Glasser song up top? Later that day I clicked on another friend listening to the same song, but it was different this time:

Wait a minute, what’s the difference? I obviously know what’s going on — the Spotify “Apply” is from a compilation while the Rdio version is on the full length release. If you click on the release version on Spotify, you can resolve the song to any other service for your friends and get recommendations, or participate in the social listening experience. But if you click on the compilation version (which is the default when typing Glasser Apply in the Spotify search box), you get nothing. The result: the song you hear might as well have been something you recorded in your basement last night, even though Rdio has what you were hoping for:

This is another common ID resolution problem. Facebook likely isn’t working from a canonical database of songs or artists, rather using loose references to them from their own data and partners. And the glue linking those ID structures together is brittle, making for risky connections and some strange user experiences when translating across services.

Why it matters

Accurate resolving is the necessary backbone of social music.

These examples show that ID resolution isn’t just the plumbing underneath a social music experience — it is the foundation of any good music service that allows sharing. If songs don’t play when they should or link to the wrong song, people can’t talk about them.

Even imagining a world in which there is only Spotify in Facebook, consider the following realities: any real music fan will want to connect with other types of services: radio players like iHeartRadio, video services like VEVO, reviews, blogs, biographies, artist photos, games, publishing platforms like Soundcloud, and so on.

We obsess about this problem. I’m guessing Facebook is obsessing about it now, because it introduces friction millions (or billions) of times per day into what Facebook wants to be a “frictionless” experience. The more social your music activity, the more you’ll agree that any decent social music service service needs to know that two slightly differently spelled artists may be the same artist. Or that the radio edit of a song can be played in place of the single version.

As implemented, the v.1 Facebook music experience is like comparing snowflakes with a ruler. Right now it impacts the user experience, but the effects could definitely get worse as more users and more services join the fray.

For me, clicking around Facebook Music these days is tough. It’s rare that anything I’d want to hear gets resolved properly:

Of course, EN knows the song exists in Rdio, as does Rdio’s API. Why doesn’t Facebook?

Artist pages and context

As you can probably tell, song resolving is bothering me enough; but the Facebook music application I was most excited about was the addition of content to its massive database of context. Facebook users or page owners can tag artist names in their wall posts and events and Facebook will helpfully make that artist playable if it knows about it:

Here Facebook made a hover link for the band tapes ‘n tapes on a page for the venue Brighton Music Hall playable by choosing a random music service that can play songs by that artist for you. This is very similar to Google’s old “Music OneBox” which aggregated MySpace, iLike, LaLa and a lot of other websites you don’t use anymore. Great for listeners, (maybe) great for services, great for the bands. But here’s another area where ID resolution problems make the user experience fall down.

It doesn’t take long to find bands that Facebook simply doesn’t “know” about, which is fascinating given the breadth and depth of their user entered and maintained “community page” and fan page structures. For example, one post down on the Brighton Music Hall page is a note about the great Dirty Beaches (Echo Nest JSON):

Where is the player? Spotify has a lot of Dirty Beaches, as do many of the services I tried. It appears that any relatively recent or independent band¹ simply does not get the player, no matter what services can support playback. This is very sad for the musicians and or listeners like me.

It’s clear and not particularly surprising that Facebook has trouble determining the identity of musicians on its own site— even those that have well groomed artist pages supplied by management with download widgets and tour details.

One of the Echo Nest rites of passage is for an engineer to uncover details automatically of the classic rock group “The Band” from the internet— and our recent push to know all we can about Facebook artists and their pages was a two month 3 engineer effort that uncovered a cascading series of Facebook pitfalls in which “The Band” was actually an easy one. One Echo Nest employee wrote me a surreal late night email asking me to make sure he was still sane as the various Facebook data gathering APIs appeared to be non-deterministic: successive calls would return completely different results². I was grimly delighted to see that Facebook’s own engineers faced the same problems we did. For some reason, it looks like we do a much better job at resolving Facebook page IDs. Again, this could definitely change as the Facebook service matures — they got a lot done in a short amount of time and Facebook music just launched. Here’s our Facebook data about “The Band” returned from a simple search for “The Band” that links to the very professional and official:

Facebook seems to want to take you here, a twilight dead letter zone of people talking about “the band” in other contexts. It’s not a page or a community page and therefore does not let cross-service resolving or context work. It looks like a SEO trap and seems to have conned 2,503 confused people:

You can see this in their music app whenever you see a band name all alone with no other information at the top. They have trouble with another favorite around here, artist resolving 101 pop quiz post-punk 80s candidate The The³:

These are not easy problems to solve. A huge class of Facebook artist resolving issues seems to come down to “merges” — artists that may be known as different names, aliases, nicknames, side projects or foreign language names. We maintain a huge database of musical aliases (“Led Zep,” “BEP” etc) as well as collaboration names and misspellings as perfect resolving against text and search is something we work very hard on. But Facebook doesn’t like seeing “Tom Jobim” because they only know him as “Antonio Carlos Jobim”:

For example, any music service worth its salt has spent countless hours debating whether to assign Tom Petty and Tom Petty and the Heartbreakers the same database ID (the answer is no, by the way.)

But when I listen to “Free Fallin’” in Spotify (Petty solo) Spotify gets it right but I am not allowed to hear it in Rdio because it doesn’t match up to their (incorrect) assignment of the song to the heartbreakers:

And this is “Free Fallin’”, a song that is taught to sixth graders and I am pretty sure is the state anthem of California. Same goes for other popular artists who’ve performed with and without named backing groups:

A real world example of this⁴: a particular dear sensitive friend in London was having a late night Jason Molina bender broadcasted on his Facebook feed. This is an inspiring use of social music and where Facebook will eventually shine— seeing what my friend is listening to, I can listen along and maybe tell him to put down the bottle of gin and go to sleep. However, I can’t hear it:

Why? Rdio had tons of Molina and of course that particular song. It’s because Facebook didn’t do a good job of resolving⁵ — to them, Jason Molina is: “Songs: Ohia,” the name of one of his side projects. And of course neither Rdio nor any other service has a song called “Get Out Get Out Get Out” by Songs: Ohia, because it doesn’t exist. This is a Jason solo song. Here’s how Facebook got so confused:

Facebook seems to rely strongly on Wikipedia for much of its artist data (their “Community Pages” are CC licensed WP copies), and Wikipedia’s editors auto-redirect Songs:Ohia to Jason. So somewhere in the depths of Facebook’s graph database is a pointer that goes the other way. This pretty much invalidates Jason’s chances of ever getting social music love on Facebook. I doubt it’ll get fixed any time soon. But maybe this one will:

That’s Selena Gomez. She’s not Facebook Music compatible, for a different reason — she has a double life. Selena is listed in Facebook as an “Actor/Director,” not a “Musician/Artist.” You can’t click through from a friend’s listen to a music service, and you can see her page from any stream activity. Can you guess why? It’s To Facebook, she doesn’t make music. This affects a lot of edge-straddling pop stars, with some notable exceptions. I noticed that “Glee Cast” was manually fixed, as was Kraftwerk (who were a “Local Business” until a month ago⁶) but comedians and musicals still are denied access to the social music party.

What could they do and what should happen

Quite a lot of Facebook’s resolving issues could be fixed by ingesting catalog in a “musical” way — not just treating strings such as artist names or song titles as database IDs as they seem to be doing. There are some pretty well-known approaches Facebook could take to fix these problems. They can use audio fingerprinting, for example (of course, I know of a couple, even an open source one.) They can work on mapping artist names and song titles together a bit more intelligently as we’ve been doing for awhile: one of The Echo Nest’s main services is project Rosetta Stone, an “ID space” resolution system that can quickly identify songs or artists in any platform: generate an eMusic ID from a Spotify URL, or a MOG ID from an audio fingerprint, or any combination possible. Facebook could have merged the data feeds in some slightly intelligent way to match songs across releases. Or some of their millions of users could edit or automatically de-reference the metadata⁷. It’s clear that none of this happened.

Social music is the future of music. Facebook is pushing this future forward more than anyone. It’s clear that Facebook has some trouble ahead of them in the resolving space and I’m sure they are obsessing about it as much as I do. They are going to have to get down with the one big database of music scenario sooner than later. Facebook has more users and data than anyone and I’d love to see a concerted effort to build a true world of connected music via the new Open Graph. But there’s a lot more work to do before that promise is realized.

- brian@echonest.com / @bwhitman

Thanks to EVB and PBL and JL and MO for editing help

⁰ Spotify actually had the nerve to call it “scrobbling,” which I doubt the inventor of the scrobble, Last.fm, are very happy about as Facebook’s integration is a clear competitor

¹ I can’t prove it yet, but I think this feature may be limited to artists appearing on an older dump of MusicBrainz— which is too bad as they only have 600,000 artists and are relatively slow to update

² Try this (failing) query for The The. You’ll either get The Bible or the Simpsons as the top result, which I found very apropos. Maybe the engs pit memcache servers against each other “I don’t like fighting either. Get here first” style

³ I’ve heard this is a Google interview question and is still to this day one of the first bands I type into a new music service after Blitter and Various Artists.

⁴ as no one really needs help listening to Tom Petty, it is somewhat an inherent instinctual property of the world that Tom Petty vibrates speakers

⁵ You can see Facebook’s resolved name if you click on a song title in Facebook Music after hovering over a music listen status

⁶ they surely have a fan in Facebook engineering

⁷ It’s likely this may be happening to some extent. If I play a song in Rdio that Facebook did not previously “know” about without Facebook’s help the next time I try it in Facebook it appears as an option. I’m not sure if this persists to other users.

The Echo Nest "puddle" and artist entity extraction

Sun, 19 Jun 2011 09:17:39 -0400

(Cross posted from my post at the Echo Nest blog, with additions)

It’s the year 2006 and co-Founder Brian and early Nest developer Ryan were trying to figure out how to associate the world of free text on the internet to musical artists. We already were crawling tens of thousands of documents a day (now millions!) but a Google-style index of unstructured text about music was not our goal. We needed to somehow quickly associate a new incoming page to an artist ID so that we could quickly retrieve all the documents about an artist as well as run our statistics on the text to find out what people were saying. Brian sketched this classic diagram, soon to be placed in the Echo Nest Museum (next to SoundCloud’s award):

That crudely drawn “blob of intelligence” that could take unstructured free text and quickly identify artist names quickly became known as “the Puddle,” a term that entered Echo Nest lore alongside “grankle” and “flat.” We use a form of the Puddle to this day. Every piece of text that our crawlers generate goes through a custom entity extraction process— it’s how we know what blogs are writing about which artists and it’s what powers our artist similarity engine, as we need to figure out what people are saying about which artists as soon as it’s said. It’s a powerful and fast changing piece of our infrastructure trying to attack a hard problem.

Entity extraction is even more useful today. If you wanted to build a Twitter app that figured out the bands a user was talking about, how could you do it? You’d need a huge database of artists (check, we have over 1.6 million), a lot of fast computers (check), and tons of rules learned from our customers over the years about artist resolution— aliases, stopwords, tokenization, merged artists and so on. Given a simple tweet:

Can we figure out what band Brian’s talking about, automatically? Well, now you can. We’ve decided to open up a beta version of our entity extraction toolkit called artist/extract to developers. Pass in any text and you’ll get back a list of artist names (in order of appearance by default, but you can sort by any Echo Nest feature) that was mentioned in the text. Think of it as a form of artist search that can take anything — Facebook comments, tweets, blog posts, reviews, SMSes.

We support all sorts of fancy things to help you. We know that “Led Zep” is an alias for Led Zeppelin. We try to deal with common word band names via capitalization rules. You can of course detect multiple artists in the same block of text. And you can use personal catalogs and Rosetta Stone to limit results to music your user owns or is playable by our partners Rdio or 7digital. And you can add the standard buckets — hotttnesss, familiarity and so on — to get information about the artists all in one call.

This is beta and still has some issues. It lags a little behind our internal entity resolving for performance reasons, and things like this can never be perfect. But it’s very helpful. Some ideas we have batting around:

Suggest bands to Facebook users using our new Facebook Rosetta service and by parsing their comments for band names
Recommend Twitter followers based on the music they talk about
Play a radio stream for any blog using our playlist APIs by parsing their posts for artist names
See how “indie” your friends are by computing average hotttnesss of all the bands they mention in email

Paul made a great demo if you want to see it in action:

Enjoy. Let us know if you have any issues!

The future music platform & music startups' imminent success

Fri, 22 Oct 2010 10:21:00 -0400

Right as I got on the plane to Amsterdam to attend the great Music and Bits conference that headlines the Amsterdam Dance Event, I was sent this article that I am sure a lot you have already read: Are music startups destined to fail? Being the co-founder and CTO of a music startup that is not failing anytime soon, I felt the urge to respond in some meaningful way. Thankfully I was about to get on stage in front of the best and brightest of Europe’s music people, so I took the opportunity to work Dalton’s warning into my talk.

Let me first correct Mashable — your headline is terrible. Amazingly, and I think this is the first and last time I’ll defend these guys but when Techcrunch is more measured than you you may want to reconsider your editorial policies. Dalton said nothing about mass failure. Echo Nest was one of their first API clients and we were about to work together before they went away, we all give each other big hugs and air kisses. He was not trying to predict my particular imminent doom. However, he was discussing something that is very hard for a lot of people and it is Echo Nest’s business to make this easy and excellent. So I made my talk about that — there is this thing happening, that I call “the music platform,” that yes, we’re working on but so are a lot of other guys, and if you are thinking of a great music experience don’t let these guys put you off. We are here to help.

The future music platform

What is a music platform?

First, we all agree that the future of the music business is apps. This is not just a party line EN takes; it is a real thing that is happening. In the past couple of years your choices as a music listener have gone up exponentially. You can hear a song on any number of mobile applications, web sites or downloadable programs. It doesn’t take a “music futurist” to point at the movement from digital downloads to interactive applications— Pandora, Spotify, Guitar Hero. Shazam & Soundhound, Hype Machine, Shuffler. Even the killer iPad game, Osmos, is really an enhanced delivery mechanism for Loscil and labelmates. There is someone now making a thing somewhere that will destroy the iTunes interface (finally) for browsing. There are a few guys pumped after a Music Hack Day that just filed papers in Delaware for a new radio app. And, yes, Google and Apple are about to throw their gilded monocles, canes and top hats into the ring any day now.

So we’ve identified three very hard things in getting these apps out there. These three put together comprise a music platform— someone to help you take care of all the annoying stuff while you concentrate on the experience.

The first one is our bread and butter, what we call “engineering.” The Echo Nest has been doing this for years — providing an outsourced music database to companies and independent devs — and we’ve gotten quite good at it. We have a database of over 10 million unique songs and 1.2 million artists, each with insane amounts of metadata. We know the pitch of the third sound of the first b-side track on James Blake’s “CMYK.” We know how many people called Spoon’s latest record “angular,” and what web sites they said it on. We can give you the tempo of any song, similar artists, loudness, a recommender and new stuff like an excellent music fingerprinter, or the stuff needed to choose concerts based on the danceability curve of the setlists. You no longer need to worry about “big data” or figuring out why Tom Petty is not the same band as Tom Petty & the Heartbreakers (ps work here if that pisses you off.) Other guys can help you with this too — the amazing Musicbrainz, of course last.fm and some other upstarts.

The second is audience— with your new experience, how can you get out to listeners? Fortunately there are very large companies competing for developers’ attention. At the rate we seem to be headed there will be an app store for combination microwave oven/toasters. But in the short term, audience from a mobile app store, social network is here now and viable. But we’d like to think it goes deeper than that— and by working with the content providers (more on that soon), we’ll be helping app developers get in front of fans with direct artist marketing.

The last, and trickiest, is content. As a music app developer, until about two months ago there was no easy way to get actual music to play without negotiating your own content deals. This is the last thing a dev wants to work on and we’ve made it our goal to help these guys out. This was the crux of Dalton’s pain, and from now on it may be helpful to consider the Echo Nest as a “proxy for the music industry.” We’ve already announced one major label deal to allow content, and we currently have two deals with digital distribution channels. With these deals in place, it takes an amazingly short amount of time to build your future vision of music. Look at this app, made in 24 hours— it is a Pandora style streaming radio app but has 10 million songs available. It uses only the Echo Nest API for all of its features. Annoying things like DMCA rules are built into our API.

Why it is awesome

I want an iPhone app that does Josh Millard’s Frasier fever dream on Prisoner episodes. I want a text message with a link to streaming audio whenever ILM flips out over a new artist. And I want imeem back, they were awesome. And with everything we’re trying I think it’s finally possible— the way we are experiencing music is changing and we want to work with brilliant people to make it change even faster. It’s not perfect yet, but we’re on the right track and I don’t think you’ll find a more focused and insane group of people in the world trying to make this work. If you’ve got ideas or questions, write me a mail.

The Echo Nest Musical Fingerprint (ENMFP)

Fri, 23 Apr 2010 23:10:52 -0400

Tomorrow begins MHD Amsterdam and at it The Echo Nest is releasing a few new things. Some of our engineering team (who deserve a severe callout for all their work, let me stick with their codenames) have been working tirelessly to get “songs” to be a first-class member of our API, and as of today, they are — we now track many millions of songs and you can query for them by name and receive all sort of useful metadata, get similar songs (with amazing results even very deep in the catalog), and even get free (legal) playable audio for a huge collection of major label content (more on this later.) As part of this push to provide data about songs, we have been working on a music fingerprint— a way to resolve an unknown audio file (what we call a “track”) to a large database to identify it in our world (as a “song.”) And we’re ready to release this to the community to see how it performs in the wild.

The design goals of our FP were to base it on Echo Nest audio features, to make it simple to implement and to make it as open as possible. Lock in of content resolution data is a terrible thing, and a large part of The Echo Nest’s focus is to make it easy for people to figure out what their music is about without getting stuck in ID space hell. If you have an iTunes collection and want to automatically make Spotify playlists, we should be able to help you. If you write an app that scans your hard drive for tracks to make great recommendations against MOG or the Limewire store, we should be able to help you. If you want the tempo of every song in someone’s terribly labeled iPod library, we should be able to help you. A fingerprint to us is a utility call— like our search_artists — a way to resolve a music identifier to our set of ID spaces. Echo Nest song IDs, if you choose to use them, give you all of our stuff “for free” — from a single EN SO ID you can get recommendations, artist pictures and bios, blog posts, record reviews, and of course all the audio analysis: the tempo, key, events in the song. But over this year we are rolling in support for any other ID space via Rosetta Stone, so you will be able to return Spotify IDs or get last.fm URLs of the song from the fingerprint. Our goal as always is to be the bridge between music and amazing applications— a platform for music intelligence that lets anyone use any service on any audio to discover and interact with music.

How it works

Our fingerprint is called the Echo Nest Musical Fingerprint (ENMFP) and is based directly on parts of our audio analysis engine that already powers tons of interactive music and music search apps across the globe. We get a detailed understanding of what is happening in a song (note: a song, not just an audio file) for “free” simply by having Tristan be our co-founder, so our work on the ENMFP started there. We worked with audio scientists on ways to scalably hash parts of the analysis and query for “codes” — a sequence of numbers that can match the same song to the ear. We identified an efficient series of transformations of our low level segment description data to make a very accurate code, and our engineering team built a suite of tests, backend servers, and a query API. The ENMFP comes in two parts. The code generator is a binary library that you can compile into your own app. It takes in a buffer of PCM samples (in practice, give it around 20 seconds of 22050Hz mono float PCM), runs a series of signal processing algorithms on the samples, and returns a list of codes. It is as simple as

    Codegen * pCodegen = new Codegen(_pSamples, _NumberSamples, offset);
    for (uint i=0;i<pCodegen->getNumCodes();i++)
        printf("%ld ", pCodegen->getCodes()[i]);

The server maintains a canonical list of songs with corresponding codes and performs fast lookup. We’ve based the server on some popular open source indexing and storage platforms, and we’ll be releasing our modifications to them as a reference implementation shortly.

Use and open nature

Almost all of this implementation is open. The data behind the server is open by design. Anyone can request full data dumps. Anyone that wants to run their own server can provided that they mirror with the other servers. The only non-normative license is in the code generator, which for now is binary-only, available for most platforms (Windows, Linux 32 & 64-bit, Mac OS X, mobile forthcoming) and free to use in any sort of application — commercial, open source, free, webapp, etc. The only pertinent restriction is that codes are sent to only “authorized servers.” The design of this license ensures that one party does not attempt to usurp the ID resolving space out from under anyone. If The Echo Nest dissolves or gets bought by a large fish cannery on accident, we want to make sure the data and query service live on without us. As a corollary, we don’t want anyone “hiding” new resolved tracks from the ID space. Anyone that collects new songs via this fingerprint has to share their data, plain and simple. This hopefully ensures that over the years the combined knowledge from all uses of the ENMFP will catalog every single piece of music available on the internet, and the data will be available to all. We want the ENMFP to grow into a public internet utility.

Features

The ENMFP looks at the underlying music, not just the raw audio signal. This gives it some unique advantages:
- Unlike many FPs, is robust to time scaling
- Can identify sample use in mixed audio
- Can identify remixes, live versions and sometimes cover versions
Can identify a song in <20s of audio
Can also match on track metadata (artist name, title, length) using Echo Nest name matching in the same call
Server and some of the code generator are completely open source
Data is completely open; dumps provided, mirroring required to host your own server (we want people to boot their own copies of the data)

Anti-features

In heavy alpha, not heavily QA’d yet, help wanted
Not completely OSS: the code generator relies on proprietary EN algorithms. Binaries provided, free to use, but not open source.
No ingestion API yet (you are querying against a large but not complete catalog, there is no way currently to add new songs. This is changing soon. If you maintain a large catalog and want it in our reference database, please get in touch.)

How to use

First, you need an Echo Nest developer API key if you don’t already have one. Next, familiarize yourself with the alpha_identify_song API. (As of right now, before we release the server source, the Echo Nest is hosting the only query server via this API.) There is instructions there on how to receive the libcodegen binaries. The libcodegen package also ships with an example code generator that you can call from the commandline, so no worries if you aren’t ready to do some compiling.

How to help

We see the ENMFP as a community project just getting started. If you are interested in booting your own mirror server, or if you have experience with FP tasks, want to help with QA, automated testing, have a large catalog to ingest or test against, please get in touch.

we are especially grateful for the work of Unrepentant Nagios Installer (UNI), Guy Who Fights With Me About the Word “Track” Every Fucking Day (GWFWMAWTEFD), Drinks Turret Coolant (DTC), Mr. HTML5 Canvas 2010 (HC2), So-Glad-I-Kept-You-Out-Of-The-Media-Lab (SGIKYOOTML), Skinny Tie (ST), Main Ontology Offender (MOO), Future Performable Employee (FPE), and of course Ben Lacker (BL)

Video

Sun, 07 Feb 2010 18:18:59 -0500