I gave a talk at Velocity EU in London a couple of weeks ago on how some pieces of the Echo Nest work, operationally:
We started Echo Nest seven years ago and I couldn’t imagine a stranger time to begin a technology-focused venture. We started with 2 self-built 2U rack machines in a closet at MIT and within two years started moving everything to first dozens and then almost thousands of virtualized hosts running on a few different cloud providers. And then a year or two ago we pulled it all back to physical again. We’re in a weird spot between offline data processor and real time API provider and none of the oft-repeated hype platforms ever worked for us.
We’ve been around before and during much of Hadoop, Solr, “NoSQL”, EC2, API-as-PR, the rise of mobile, Apple on Intel, Hacker News and sharded mySQL as a key-value store. And we’re still at it, making money in an industry that would rather keep it to itself. If it’s not obvious, I’m really proud of what we built and the small team that works their ass off to keep it working while making some really cool new stuff.
There’s a lot of stories to tell and this is just the first one, on how we’ve abused text indexing to do quite a lot of our work. If you ever wondered how some of the musical data souffle avoids burning, here it is. Thanks to John & Kellan for giving me a chance to present this.