¶ We Will Know Ourselves by Our Love; We Will Know You By What You Let Go · 1 February 2024 listen/tech
Collective listening is a cultural investment. Collected listening data can be valuable for music streaming services' selfish business purposes, of course, but it's generated by music and listeners, and should be valuable to the world and to music first.
It was my job, for a while, to try to turn music-listening data into cultural knowledge. My opinion, from doing that, is that there are four fundamental kinds of socially valuable music-cultural knowledge that can be learned, with a little attentive work but no need for inscrutable magic, from listening.
The first is popularity. The most fundamental change in our knowledge about music and love, from the physical era to the streaming era, is that we now know what every listener plays, instead of only what they buy. In its simplest form this produces playcounts, and thus the most basic form of streaming transparency and accountability is showing those playcounts. Streaming services have to track plays for royalty purposes, obviously, but music accounting is done by track, and cultural accounting is done by recording and song. At a minimum, we consider the single and the reappearance of that same exact audio on the subsequent album to be one cultural unit, not two, and thus want to see the total plays for both tracks combined in both places. Most major current services do this adequately, albeit at different levels of precision (and one major service glaringly does not display playcounts at all). But really, as people we know that the live version of a song is the same song as the studio version, and if we ask each other what the most popular song on a live album is, we do not mean which of those literal live recordings has been played the most, we mean which of those compositions has been conjured into the air the most across all its minor variations. So far no service has attempted to show this human version of popularity in public, although probably all of them have some internal representation of the idea for their own purposes. (I have worked on various logistical and cultural issues around song identity and disambiguation over the course of my time in music data, but never on the actual mechanics of music recognition, ala Shazam.)
The second kind of knowledge, derived from the first, is currency. We would like to know, I think, what music people are playing "now". Ariana Grande's new song is currently hotter than her old ones, even though it is nowhere near the total playcount of the old ones yet. This can be calculated with windows of data-eligibility, or by prorating plays by age, and most major services do some version of this, but only share it selectively. Spotify, for example, uses an internal version of currency to select and rank an artist's 10 most "Popular" tracks, but only those 10, and the only numbers you actually see there and elsewhere in the app are the all-time playcounts. I worked on a currency algorithm at the Echo Nest, before we were acquired by Spotify, but it's hard to do this very well without actual listening data, and the one Spotify had already devised from better data, without us, produced better results without being any more complicated.
The third kind of knowledge, moving a big step beyond basic transparency, is similarity. Humans listen to music non-randomly, and thus the patterns of our listening encode relationships between songs and between artists. Most current services have some notion of song similarity for use in song radio and other song-level recommendations, and also some notion of artist similarity for behind-the-scenes use in artist radio and more explicit use as some kind of exploratory artist-level navigation ("Related Artists", "Similar Artists", "Fans Also Like", etc.).
I worked on multiple generations of these algorithms in my 12 years at the Echo Nest and then Spotify, and as of my departure in December 2023 the dataset for the "Fans Also Like" lists you see on artists pages in the Spotify app was my personal work. In my time there I had many occasions to compare competing similarity algorithms, both in and out of music, and in a better world less encumbered by petty confidentiality clauses, I would cheerfully bore you with the tradeoffs between them at great length. In my experience simple methods can always beat complicated methods because they're so much easier to evaluate and improve, and time spent refining the inputs is usually at least as productive as tweaking the algorithms themselves, but much less appealing in engineering terms. I consider the calculated similarity network of ~3 million Spotify artists, as I left it, to be a historically monumental achievement of collective listening made mostly possible by streaming itself, but having had to do a lot of internal lobbying on behalf of the musical cogency of similarity results over the years, I am forced to concede that my personal stubbornness is more relevant than any one individual ought to be in this process. Spotify still has my code, but stripped of my will and belief I'm not sure it will thrive or even survive. My individual layoff doesn't necessarily express a Spotify corporate opinion on any larger subject, but it's hard to deny that if Spotify cared, organizationally, about giving the assisted self-organization of the world's listening back to the world, my individual production role in this specific form of it would have been a trivial and uncontroversial excuse for not letting me go. If they give up on this whole feature as a result of one person's absence, it will be a tragic and unforced loss for everybody.
The fourth key form of music knowledge, moving up one more level of abstraction from pairwise similarity, is genre. Genres are the vocabulary by which we understand and discuss music, and genres as communities are the way in which music clusters together in the world. Genres are communities of artists and/or listeners and/or practice, and usually some combination of all three. AI music will be meaningless and inherently point-missing if it attempts to apply sonic criteria without any references to communities of creation or reception, and it will turn out be just one more non-scary new tool in the long history of creative tools if it ends up rooted in how communities sing to themselves about their love. There is no "post-genre" music future, or at least no non-nihilistic one, because music creates genres as it goes.
There are three ecosystemic ways to approach the data-modeling of musical genres: you can let artists self-identify, you can crowd-source categorization from listeners, or you can moderate some combination of those inputs with human expertise.
Two of those ways don't work. Artists self-identify aspirationally, not categorically. If you try to make a radio station of all the rappers who describe themselves as simply "hip hop", you will get a useless pool of 75,000 artists from which most will never be selected. Listeners, conversely, describe music contextually, so two different listeners' "indie pop" playlists may be using the phrase "indie pop" in totally unrelated ways, and thus may have no cultural connection at all. But motivated humans, especially if they know some things about music and are willing to learn more, can mediate these difficulties and channel noisy signals into guided and supervised extrapolations.
You might expect that a global music-streaming service, in recognition of its dependence on music and thus its responsibility to steward music culture, would have a large dedicated team working constantly on systematic, culturally-attuned genre-modeling. Spotify did not. It had editors making playlists, which is sometimes a form of genre curation and sometimes is not. It had ML engineers trying to find correlations between words in playlist titles and tracks, despite playlist titles very much not being a track-tagging interface at all, never mind a genre-categorization tool. It had a handful of people doing specific genre-curation work, mostly on our own initiative because we knew it was worthwhile. And it had me maintaining the genre system, with all its algorithms and all its curation tools. I invented the system (at the Echo Nest, before we were even acquired), I ran it, I supervised it, I tweaked it, I defended it, I believed in it, I helped people apply it to other music and business problems. I had a Slack trigger on the word "genre", so you could summon me from anywhere in Spotify by just typing it. The system grew from hundreds of genres to thousands. My own personal site, everynoise.com (which also predated the Spotify acquisition), was a way to share a sprawling holistic view of it that would never have made sense inside a black-and-green Spotify window or even a white Rdio window before that. I never managed, in ten years of trying, to get genres integrated into the actual daily Spotify music experience (I wanted there to be a list of Fans Also Like genres on artist pages right under the list of Fans Also Like artists; both of these are forms of cultural context and collective knowledge), but I know, from years of emails and stories and other people's independent enthusiasm (including, only shortly before the layoffs, this one in The Pudding, which said "an always-updating catalog of 6,000 genre is groundbreaking" with unfortunate foreshadowing) that I wasn't the only person who understood the value of this whole earnest and unruly and seemingly-endless project.
Will I be proven wrong about the "endless" part? Here, again, we cannot simply conclude that Spotify does not care about genres and music culture because I got laid off. The code remains. Some of the other people who did genre-curation work are still there. Spotify could just keep the internal system running, even if nobody but me would have the inclination or expertise to improve it any further. And maybe they will. I hope they will. It doesn't cost much in computing terms. Spotify is the world's most dominant music-streaming service and genres are how music evolves and exists. Surely one cares about the other.
But if they cared, and one person in a still-8000-person company is basically the smallest practical unit of care, keeping me around would have been self-evidently worthwhile. The genre system wasn't even the only thing I did. The genre system and Fans Also Like weren't even the only things I did. The genre system and Fans Also Like and Wrapped weren't even the only things I did. The public toys I made were the tiniest fraction of my work. If everything I did do wasn't enough, maybe they don't care, and maybe all these things will be unceremoniously abandoned.
But what comes from us, and is made out of our love, of course we can and will rebuild over and over. Spotify is not the only collector of collective listening. These were not the first attempts to connect artists through their shared fans, or to model the genres into which we assemble, and they were never going to be the last. Maybe we will look back on these meager, patchwork networks of only 3 million artists, and only 6000 genres, like we keep the absurdly self-important book reports our kids wrote when they were 9. We are proud of their care and their ambition, not their page-counts. We remember what they dreamed of becoming, and then we hug the people they are in the midst of becoming, and then we think about what we are going to do and become tomorrow.
It was my job, for a while, to try to turn music-listening data into cultural knowledge. My opinion, from doing that, is that there are four fundamental kinds of socially valuable music-cultural knowledge that can be learned, with a little attentive work but no need for inscrutable magic, from listening.
The first is popularity. The most fundamental change in our knowledge about music and love, from the physical era to the streaming era, is that we now know what every listener plays, instead of only what they buy. In its simplest form this produces playcounts, and thus the most basic form of streaming transparency and accountability is showing those playcounts. Streaming services have to track plays for royalty purposes, obviously, but music accounting is done by track, and cultural accounting is done by recording and song. At a minimum, we consider the single and the reappearance of that same exact audio on the subsequent album to be one cultural unit, not two, and thus want to see the total plays for both tracks combined in both places. Most major current services do this adequately, albeit at different levels of precision (and one major service glaringly does not display playcounts at all). But really, as people we know that the live version of a song is the same song as the studio version, and if we ask each other what the most popular song on a live album is, we do not mean which of those literal live recordings has been played the most, we mean which of those compositions has been conjured into the air the most across all its minor variations. So far no service has attempted to show this human version of popularity in public, although probably all of them have some internal representation of the idea for their own purposes. (I have worked on various logistical and cultural issues around song identity and disambiguation over the course of my time in music data, but never on the actual mechanics of music recognition, ala Shazam.)
The second kind of knowledge, derived from the first, is currency. We would like to know, I think, what music people are playing "now". Ariana Grande's new song is currently hotter than her old ones, even though it is nowhere near the total playcount of the old ones yet. This can be calculated with windows of data-eligibility, or by prorating plays by age, and most major services do some version of this, but only share it selectively. Spotify, for example, uses an internal version of currency to select and rank an artist's 10 most "Popular" tracks, but only those 10, and the only numbers you actually see there and elsewhere in the app are the all-time playcounts. I worked on a currency algorithm at the Echo Nest, before we were acquired by Spotify, but it's hard to do this very well without actual listening data, and the one Spotify had already devised from better data, without us, produced better results without being any more complicated.
The third kind of knowledge, moving a big step beyond basic transparency, is similarity. Humans listen to music non-randomly, and thus the patterns of our listening encode relationships between songs and between artists. Most current services have some notion of song similarity for use in song radio and other song-level recommendations, and also some notion of artist similarity for behind-the-scenes use in artist radio and more explicit use as some kind of exploratory artist-level navigation ("Related Artists", "Similar Artists", "Fans Also Like", etc.).
I worked on multiple generations of these algorithms in my 12 years at the Echo Nest and then Spotify, and as of my departure in December 2023 the dataset for the "Fans Also Like" lists you see on artists pages in the Spotify app was my personal work. In my time there I had many occasions to compare competing similarity algorithms, both in and out of music, and in a better world less encumbered by petty confidentiality clauses, I would cheerfully bore you with the tradeoffs between them at great length. In my experience simple methods can always beat complicated methods because they're so much easier to evaluate and improve, and time spent refining the inputs is usually at least as productive as tweaking the algorithms themselves, but much less appealing in engineering terms. I consider the calculated similarity network of ~3 million Spotify artists, as I left it, to be a historically monumental achievement of collective listening made mostly possible by streaming itself, but having had to do a lot of internal lobbying on behalf of the musical cogency of similarity results over the years, I am forced to concede that my personal stubbornness is more relevant than any one individual ought to be in this process. Spotify still has my code, but stripped of my will and belief I'm not sure it will thrive or even survive. My individual layoff doesn't necessarily express a Spotify corporate opinion on any larger subject, but it's hard to deny that if Spotify cared, organizationally, about giving the assisted self-organization of the world's listening back to the world, my individual production role in this specific form of it would have been a trivial and uncontroversial excuse for not letting me go. If they give up on this whole feature as a result of one person's absence, it will be a tragic and unforced loss for everybody.
The fourth key form of music knowledge, moving up one more level of abstraction from pairwise similarity, is genre. Genres are the vocabulary by which we understand and discuss music, and genres as communities are the way in which music clusters together in the world. Genres are communities of artists and/or listeners and/or practice, and usually some combination of all three. AI music will be meaningless and inherently point-missing if it attempts to apply sonic criteria without any references to communities of creation or reception, and it will turn out be just one more non-scary new tool in the long history of creative tools if it ends up rooted in how communities sing to themselves about their love. There is no "post-genre" music future, or at least no non-nihilistic one, because music creates genres as it goes.
There are three ecosystemic ways to approach the data-modeling of musical genres: you can let artists self-identify, you can crowd-source categorization from listeners, or you can moderate some combination of those inputs with human expertise.
Two of those ways don't work. Artists self-identify aspirationally, not categorically. If you try to make a radio station of all the rappers who describe themselves as simply "hip hop", you will get a useless pool of 75,000 artists from which most will never be selected. Listeners, conversely, describe music contextually, so two different listeners' "indie pop" playlists may be using the phrase "indie pop" in totally unrelated ways, and thus may have no cultural connection at all. But motivated humans, especially if they know some things about music and are willing to learn more, can mediate these difficulties and channel noisy signals into guided and supervised extrapolations.
You might expect that a global music-streaming service, in recognition of its dependence on music and thus its responsibility to steward music culture, would have a large dedicated team working constantly on systematic, culturally-attuned genre-modeling. Spotify did not. It had editors making playlists, which is sometimes a form of genre curation and sometimes is not. It had ML engineers trying to find correlations between words in playlist titles and tracks, despite playlist titles very much not being a track-tagging interface at all, never mind a genre-categorization tool. It had a handful of people doing specific genre-curation work, mostly on our own initiative because we knew it was worthwhile. And it had me maintaining the genre system, with all its algorithms and all its curation tools. I invented the system (at the Echo Nest, before we were even acquired), I ran it, I supervised it, I tweaked it, I defended it, I believed in it, I helped people apply it to other music and business problems. I had a Slack trigger on the word "genre", so you could summon me from anywhere in Spotify by just typing it. The system grew from hundreds of genres to thousands. My own personal site, everynoise.com (which also predated the Spotify acquisition), was a way to share a sprawling holistic view of it that would never have made sense inside a black-and-green Spotify window or even a white Rdio window before that. I never managed, in ten years of trying, to get genres integrated into the actual daily Spotify music experience (I wanted there to be a list of Fans Also Like genres on artist pages right under the list of Fans Also Like artists; both of these are forms of cultural context and collective knowledge), but I know, from years of emails and stories and other people's independent enthusiasm (including, only shortly before the layoffs, this one in The Pudding, which said "an always-updating catalog of 6,000 genre is groundbreaking" with unfortunate foreshadowing) that I wasn't the only person who understood the value of this whole earnest and unruly and seemingly-endless project.
Will I be proven wrong about the "endless" part? Here, again, we cannot simply conclude that Spotify does not care about genres and music culture because I got laid off. The code remains. Some of the other people who did genre-curation work are still there. Spotify could just keep the internal system running, even if nobody but me would have the inclination or expertise to improve it any further. And maybe they will. I hope they will. It doesn't cost much in computing terms. Spotify is the world's most dominant music-streaming service and genres are how music evolves and exists. Surely one cares about the other.
But if they cared, and one person in a still-8000-person company is basically the smallest practical unit of care, keeping me around would have been self-evidently worthwhile. The genre system wasn't even the only thing I did. The genre system and Fans Also Like weren't even the only things I did. The genre system and Fans Also Like and Wrapped weren't even the only things I did. The public toys I made were the tiniest fraction of my work. If everything I did do wasn't enough, maybe they don't care, and maybe all these things will be unceremoniously abandoned.
But what comes from us, and is made out of our love, of course we can and will rebuild over and over. Spotify is not the only collector of collective listening. These were not the first attempts to connect artists through their shared fans, or to model the genres into which we assemble, and they were never going to be the last. Maybe we will look back on these meager, patchwork networks of only 3 million artists, and only 6000 genres, like we keep the absurdly self-important book reports our kids wrote when they were 9. We are proud of their care and their ambition, not their page-counts. We remember what they dreamed of becoming, and then we hug the people they are in the midst of becoming, and then we think about what we are going to do and become tomorrow.