When a storm approaches, make sure you have sufficient essential supplies: coffee, cereal, soup, the ingredients for making pancake batter from scratch, cats, enough paper to fold a card and an airplane for every person you know, strobe lights, a 5-hour playlist that eventually evolves from frenetic drum & bass into whatever you hope to feel after all of this is done.

At work I have this thing where you can pick any band and pull on the strands that connect them to the rest of the universe of music. This is what happened the other day when I wanted to see where the excellent new Kate Boy song "Northern Lights" took me. Bits of fragile atmospheric sparkly electronica started glittering out and there kind of doesn't seem to be any way to make them stop.  

I basically do this kind of thing on and off most work days, and yet nearly every time, even with bands I found this way to begin with, I end up with dozens more interesting songs by bands that I'd never even encountered. I don't think the code is creating the bands out of nothing by its very invocation, but if I hadn't written the code I don't think I'd be so sure.

In the course of checking something at work yesterday I serendipitously discovered that one of our founders has a deeply personal connection to Christian indie progressive rock, as of course do I. Examples of this sadly unappreciated form can be vexingly hard to find, so I made a little playlist of relevant discoveries to which we can refer if we ever need a spiritual boost. Note that not all of these bands are avowedly Christian, but I think we can agree that this music is doing the Lord's work whatever its ostensible affiliations.

The 2012 Village Voice Pazz & Jop music-critics poll is now out. As has been the case for the last few years, I did the data-correction and tabulation for it. The results of this work are here:  

· The Voice's P&J home
· My piece about statistics and analyses
· My massive stastistical hyperindex to the last 5 years of the poll
· My own ballot  

The short version: Frank Ocean, Carly Rae Jepsen. If the former of these surprises you, you probably aren't paying attention to music criticism. If the latter, you probably aren't paying attention to pop. Neither of these are life flaws, mind you. But now you have a chance to catch up.  

PS: This poll used to be tabulated by people. This is not what people are good at. Eventually computers were employed to help count, but cleaning up the data so it could be counted by computers still took several person-weeks of unpleasant human effort. Then some coworkers and I spent in the neighborhood of 25 person-years building a data-correction and -analysis system with which the whole thing could be done in about 3 person-days. Then Google bought the company for which we made that thing, and shut it down. So I spent about 4 person-days writing a new correction/analysis system from scratch myself, with which this year's poll took about 4 hours to correct and tabulate. Including fixing a few stray errors that the previous system missed.  

There is a moral in there somewhere.

I work at a music company. Our holiday party involves people who work there performing music. I occasionally make music, but I almost never actually perform it. Very, very close to never.  

But I wanted to this year. So I wrote a new song to sing. It's called "The Rules That Govern Hearts". I have a couple weeks to practice singing it, but you can start practicing listening to it right now.  

The Rules That Govern Hearts (3:37)

Somewhere around 1992 or so, I had to write my "personal goals" at work for the first time. I put down some stuff about design and usability, which I have long since forgotten, but I distinctly remember that the last item was "Exploit the power of information technology to improve my music collection."  

My job, at the time, had nothing to do with music. It did provide me with an internet connection for downloading Gary Numan discographies from Usenet, and it did once send me to a UI conference in Amsterdam, from which I came back with approximately 130 pounds of European CDs. But the connection from what I was doing to what I was listening to remained a little abstract.  

But that job got me the one after that, which got me the one before this, which got me to now. The scope of "music collection" has expanded quite a bit over this time, obviously, but in fact I am very literally now paid to exploit the power of information technology to improve the world's experience of music.  

The vast majority of things I work on, to this end, involve identification or disambiguation or extrapolation or interpolation. They are corrective or exploratory or contextual. They try to carry you from somewhere to somewhere else, or to rescue you from something squelchy along the way. They try to get computers to understand enough about human cues and contexts and constraints to fill humanless spaces with rough facsimiles of what humans would have suggested if they'd had the time.  

But I just did one that isn't so much like this. One of the computed song metrics I've been working on is Discovery, which attempts to quantify the idea of songs emerging. This is a detection metric, not an aesthetic one. It isn't trying to tell you songs you should listen to, it's trying to find the songs that people are in fact starting to listen to more than the established prominence of the artists can explain. Discovery songs tend to be new, but a fair number of songs get their unexpected breaks after they've already been out for a year or two. And once enough people discover a Discovery, obviously, it stops qualifying.  

There are a lot of fairly arbitrary thresholds and weights involved in this calculation. How much prominence constitutes critical mass, and how much constitutes overexposure? How old can something be and still be sorta new? And the notion is inherently subjective, of course: anything you've heard a dozen times yourself is no longer a discovery to you, whereas a song 1.3 billion people have been dancing to across Asia every morning for the last 6 months still could be. And there is no way to stipulate correct answers against which this score can be quantitatively tested.  

But provable correctness is overrated, or at least sometimes irrelevant. Music discovery is working if you can use it to discover music. And now you can test this particular method yourself. Starting today I'm going to be maintaining an official Echo Nest playlist on Spotify with the top 100 songs of the moment according to this discovery score. It's here:  

The Echo Nest Discovery  

If you subscribe to it, each week's new songs will get flagged for you automatically.  

The songs the robots find cheerfully comply with no particular style or pattern. The set is not coherent in any human sense. It's in rank order, but the ranking logic will not be evident or audible. There will be interminable dubstep remixes next to country laments next to Christian hardcore next to Azorean folk traditionalism. You should expect to hate half of these, and find half of the rest inscrutable. This is definitively impersonal. Somebody, somewhere, liked each of these songs, but we've stripped them of any idea of whom or why.  

And yet, the fact that you don't know those people doesn't mean you won't agree with them. I endorse this discovery experiment on the purely empirical grounds that I have personally discovered things this way myself. So, for the record, I'm also going to maintain a personal playlist of what I liked from the robot one:  

Treasures the Robots Brought Me  

My suspicion, which you're welcome to evaluate for yourself, is that my human picks will be essentially no less random for you than the overall robot list. But if you find something, either way, we both win.  

[I'm not sure how the robots win. Probably they win no matter what. If I have a non-musical goal to add at this job, it's probably to keep trying to make sure the human/robot thing is never zero-sum...]

Just as night begins to fall, you come across some people. They're cold, and they're scared, but they've made a pile of wood and they're trying to start a fire.  

The Salesperson's Dilemma is: How much can you charge them for matches?  

The Businessperson's Dilemma is: If you give them some matches for free tonight, can you sell them new axes in the morning?  

Then they get the fire started.  

And now the Entrepreneur's Dilemma is: Can you persuade them to try the bizarre experiment of rearranging the pile of wood into a big hollow box, called a "house", faster than they can burn the rest of the wood?

I have a post up on the company blog at work today. It's about music or math, depending on your perspective.

We will look back on these days, I think, as some weird interlude after the invention of computers but before we actually grasped what they meant for us. The Age we are stumbling towards, I am very sure, is the Age of Data. And when we get there, we will be there because we have sublimated the state-machine mechanics of computers beneath the logical structural abstractions of information and relation, and begun to inhabit this new higher world without reference to its substrate.  

I spent 5 years of my life trying to help bring this future about. That is, in a sense I've spent my whole adult life trying to help bring this future about, but for those 5 years I got to work on it very directly. I designed, and our team built, an attempt at a prototype of what a new data exploration system could be like, and at the core of this was my attempt at a draft of a language for discussing data the way algebra is a language for discussing math. These are the elements out of which this new age's alchemies will be constituted. And there were moments, as the system began to come into its own, when I felt the twitches of power awakening. You could conjure shapes out of data with this thing. It made information malleable, made it flow.  

The computer programmers on the team sometimes referred to the project as a system for "non-programmers", and I've come to think of that as both its potential and its downfall. Programmers never say "non-programmers" as a compliment. At best it's merely condescending, at worst it's a euphemism for "idiot" or a semi-aware admission of incomprehension. For programmers, programming is by definition an end, not a means, and therefore the motivations of non-programmers are inherently mysterious and alien. But what we built was for non-programmers in the same way that a bridge is for non-engineers. That is, the whole point of it was to represent a different interaction model between people and information than the ones offered by, at one end, programming languages, and at the other spreadsheets and traditional database programs. As I said over and over throughout those 5 years, I was trying to get us to do for hyper-connected datasets what VisiCalc once did for columns of numbers. I wasn't trying to simplify; if anything, I was making some things harder, or at least less familiar. This new age is not a subset of a previous age. It is not for lesser people, and its challenges are not of a simpler character.  

And as Google now shuts that system down, literally unceremoniously, and 5 years of my work and dreams and visions are at least nominally obliterated, I feel a little sadness but mostly relief. I'm still very convinced that our tools -- humanity's tools -- for interacting with data are hopelessly primitive. I'm still convinced that it won't make a whole lot of difference what those tools are if kids don't grow up learning how to think about data in the first place. I'm still convinced that I have a blurry, fractured vision of what it might take to change these things.  

But I also realize two more things.  

First, the system we built was only a beginning, and it had hardened into a premature finality long before its official corporate fate was settled. The query language I invented was cool, but the successor to it, which I'm sketching in my head whether I want to or not, is a different sort of thing yet again. And I was never going to reach it incrementally, arguing over every syntax decision on the way. Sometimes you have to just start over. The next one will not aspire to be the Visicalc of anything. It's not better business tools we need. The problem is not that we are alienated from our inner accountants. The thing we need first is not even an algebra of data, probably, but an arithmetic of data. We need an inversion of "normalization" in which you don't write data wrong and then endure six Herculean labors to make it obscurely more pleasing to capricious gods, but rather a way of writing it in the first place with an inherent expressive gravity towards truth because more true is always more powerful. This is a task in applied philosophy, not programming and not engineering and not even science. We need to imagine what Plato would have done when his record collection got too big for his cave.  

Second, I still believe that we all deserve better tools, tools more suited for our actual tasks and needs as people whose lives and choices and options are increasingly functions in, not merely of, information. But in the process of exploring what I mean by that I've become a non-non-programmer myself. At my new job I am an engineer. And sometimes, when you think you know what the better world looks like, you can bring pieces of it up out of your dreams. You can walk where the new paths will be. With enough belief, you can walk where the bridges will be. I will come back to these paths, one way or another, but you never do great things by imagining what people you don't understand might want for purposes you don't grasp or embrace. You should trust your own judgment only where you love beyond reason. Anybody could do nearly anything with Needle, and the business cases for it all involved hypothetical big companies doing hypothetical big things with hypothetical big data that repeatedly never actually materialized (and might have been hypoethical if they had). But left to my own invented devices, I always ended up using it for music data.  

So I have followed my own love, and my own obsessions, deeper into that data. At my new job, I am trying to make sense of the largest music database in the world, which is a lot more fun than what I was doing before, and harder, and of rather more direct and demonstrable relevance to anything. On my own, I will continue the music projects I started in Needle. The Discordance evolved out of empath, and so I've evolved it back in, with less marginalia but maybe more coherence. For the Pazz & Jop I've built a stats site far more specific than I could ever have done in the generalized environment of Needle. These will grow as I play with them, and probably there will be other things. I spent 5 years trying to build fancy tools, but it's pretty amazing what you can do with just a hammer. I was Needle's most dedicated user, but in the end, both sadly and happily, I don't actually need it any more. Nobody will miss it more than I will, but maybe nobody will really miss it very much. The moral, I think, and maybe even the ethic, is that these systems do not matter. This isn't the first system I worked on only to see it shut down, and it won't be the last. Software is the epitome of ephemera, necessary in aggregate but needless in every mundane specific.  

But the things we learn from these systems stay learned. Even the ways of learning remain ways after their original demonstrations disintegrate. This is another phrasing of the point about this Age, in fact: the flow from Data to Information to Knowledge to Wisdom is not a function of syntax or platforms or prevalence or virtualization. It is something we do, to which the technology is merely witness. We must teach our children how to think about data because the data survives where the systems fail. We must teach ourselves to be children again in this new Age, because its most transformative truths still await discovery, and are anything but mundane or needless, and we will never recognize them unless we can recall what it felt like in our hearts when everything was amazing and new and ahead of us, and the act of waking was an invitation to wonder to show us a way.