The Long Tail

Who says TV has to come in 30-minute multiples?

UPDATE (see below)

I have a bet with a friend that within ten years most TV will no longer come in 30-minute chunks. When you think about it, there's nothing magical about half hours; they're simply an easy way to divide a broadcast programming schedule into segments that start and finish on the hour. Outside of the broadcast schedule, entertainment and news comes in all sorts of lengths, from 30-second clips to three-hour concerts; there's no premium on thirty minutes.

Like so many other conventions that we today accept as cultural choice, the rigid programming convention of making video in multiples of 30 minutes is actually an artifact of inefficient distribution. I think it's going to eventually fade away, replaced by a range of more natural lengths of video content that reflect the diversity of human attention spans and content types, not network programming convenience and advertiser priorities. This is yet another example of the sometimes surprising implications of the shift from scarcity to abundance in distribution; it's also an example of how ingrained scarcity thinking is in our culture.

So what are those more natural lengths of video content? Well, when I look at our own family's video watching, virtually none of it is in half-hour chunks. I don't watch much TV, but I do watch random web video stuff (1-10 minutes) and movies (1.5-2 hours). Our kids watch TV shows on the DVR but they're trained to skip the ads, so video for them comes in 20-minute bites. My wife watches most of her favorite TV series on DVD, which can lead to an evening of anything from an hour to three hours.

My sense is that most video wants to be shorter than 30 minutes. At the moment, it's anyone's guess, so I did a little research to quantify what I could. The results are shown in the chart below. The red bars represents the statistical profile of our current cable programming, minus some uncategorizeable material (HSN, CSPAN). The blue bars are mostly guesswork on my part, informed somewhat by the average lengths of video programming found on bittorrent.

Some observations: I was surprised to find that there's actually more TV today in one-hour chunks than in half-hours (at least that was the case in my analog cable sample). The two-hour bar is movies, but the 2.5 and 3 hour bars are mostly sports. Different times of day have different ratios of half-hour and one-hour programming (mostly 30-minute sitcoms in the early evening and 30-minute kids shows in the morn; hour-long talk shows take over in midday and the later evening). The big picture, however, is all too predictable: almost all current TV programming exists in 30-minute multiples.

In the IP-TV future, where video is pulled on demand from anywhere, I suspect we'll be watching more and more shorter stuff. We're already channel grazing, jumping from one sub-minute video sample to another, so we're clearly comfortable with short stretches of video, even if that's not the way it's supposed to be watched. Why not acknowledge the reality and offer TV in naturally attention-deficit lengths for a generation that's going to watch it that way regardless? Likewise, TV news shows are assembled by stringing together dozens of short items over the course of 30 minutes or an hour; wouldn't it be even better to let viewers pick the ones they want to watch from a menu instead?

Rather than today's peak at 30 minutes, I think we'll see smaller peaks at 1, 2, 10 and 20 minutes (shown). Elsewhere, the time demand curve will smooth out a bit as more shows break the tyranny of the 30-minute multiple. Sports, in particular, could be sliced into dozens of new lengths: full games, highlights, key quarters/innings, last two-minutes, and so on. It's already that way on the web; I suspect it won't be long before TV goes the same way.

(By the way, all these points about the Long Tail changing the size and shape of media itself apply equally well outside of TV. See this post about the same in articles, books and encyclopedia entries.)

UPDATE:

I just noticed a great essay from Mark Pesce that describes the effects of "hyperdistribution" on TV, including the move to shorter programming:

The television moment is becoming more pervasive, as television  spreads into mobiles and laptops and game machines. This is creating an enormous demand for programming well-suited to these devices and the situations where they're commonly used. This is the archetypal example of someone waiting for a bus or train, or having a few spare minutes at lunch. This audience often doesn't have the time to watch a 22-minute or 44-minute program; they have a few minutes to spare, and want to be taken out of the moment.

This market generally favors comedy - such as Cartoon Network's "Adult Swim" episodes, which run for 11 minutes, or even shorter pieces, such as "Happy Tree Friends," or JibJab's "This Land" (which had over 70,000,000 downloads in one week). It seems that the shorter and funnier the piece, the further it is likely to travel. That said, this doesn't mean that television is about to devolve into slapstick. Robot Chicken [an Adult Swim show], for example, is often highly intelligent, with jokes that work on several levels simultaneously, including satire, parody, and slapstick.

When you think about it, there's plenty of precedent for this size change in other media. Once written news came in newspaper-article-sized chunks each morning. Now it comes in an infinite supply of screen-sized chunks, published all through the day, just as soon as news happens. Newspapers, as a result, are declining as the marketplace shifts its demand from modestly authoritative news on paper 12-24 hours after the fact to modestly authoritative news on screen 10-20 minutes after the fact. For substance, meanwhile, there are still books and monthly magazines. The market, in other words, is abandoning the middle: people increasingly want both short now and long next month, but not 12 column-inches tomorrow.

I think that the 30-minute show is the newspaper of television--a format born of distribution scarcity that is now past its primetime. Demand will shift to shorter content for convenience and entertainment, and longer content for substance and satisfaction. But the middle will not hold.

Being a cliché is a *good* thing, right?

Thanks to David Hornik for this, which comes from an actual venture capital pitch presentation.

As it happens, IODA actually is a great Long Tail example. It stands for the Independent Online Distribution Alliance, and they make it easy for independent music artists to get their songs on iTunes and the other digital music services. These services don't want to have to deal directly with thousands of independent labels, and the labels (often just a single band's business side) typically have no idea how to work with the services.

You can think of companies like IODA, CDBaby and The Orchard as rights brokers who bridge the impedance mismatch between the commercial and non-commercial worlds. As such, they serve an essential role in connecting heads and tails. By letting Long Tail producers plug into Long Tail content aggregators, these brokers help populate the tail in digital music services. Once tail content is seamlessly integrated with head content, recommendations and other filters can drive demand from one to the other, revealing heretofore underestimated markets for niches.

A rocket launcher of our own

The object to the right may not look like much, but it's one of the more mind-blowing things I've encountered. You know how the worst Hollywood cliche of bad cybermovies is that some kid is playing a videogame and he gets SUCKED INTO THE SCREEN and then must fight game baddies for real? Well, what if that could happen in reverse? What if you could turn videogame vehicles, characters and scenes into real objects?

Now you can. Here's how.

I'll start by letting reBang, where I discovered the picture, explain it:

The image is a screen capture from Pro/ENGINEER CAD, perhaps the most widely used product development 3D application for design and manufacturing. That object is a piece of a virtual game object “captured” from id’s Quake 3 videogame (the barrel of a Rocket Launcher). It was not created in my CAD application. It was not ripped from the game files. I “hijacked” the data streaming to my monitor using a freely available tool. And now, if I desired, I could manipulate the data and create a real product.

Let's pause for a moment to let that sink in. What reBang is describing is nothing less than a three-step process for turning cool bits into even cooler atoms:

1. Capture a scene from your favorite videogame.
2. Import it into a CAD program and isolate an object you'd like to have.
3. Send it to a fabricator, either a 3D printer or a computer-controlled milling machine, and watch it emerge as a physical object.

Presto--your own rocket launcher, just like those in Quake.  It won't work, but it will look just right (especially after you paint it). Why wait for some toymaker to get around to making an action figure of your favorite videogame character? Do it yourself!

I think I will. Inspired by a Neil Gershenfeld fablab speech, I plan to eventually convert part of our garage into into the coolest screen-to-steel battlebot factory around. Right now we're still in the Lego league, but I figure that the way to win the kids over to proper DIY matter hacking is to start by magically turning their fave virtual characters into real ones.

reBang doesn't describe the process, but here's what my research has turned up. That "freely available tool" that captures 3D coordinates from videogames? I think it's HijackGL, which intercepts a videogame's OpenGL calls to the video card and outputs the geometry data in standard 3D CAD formats.

We'll edit that data in Blender, an open source CAD program. Then, since we're not quite ready to buy our own 3D printer yet (even on eBay), we'll probably send it off to be fabbed at our local 3d printing shop.

That's good enough for toys. But for real battlebots, we're going to want to make individual parts out of metal and plastic, and for that we'll want to move to a proper computer-controlled milling machine. Right now a Roland MDX-15 is less than $3,000 on eBay, so as soon as the youngest of the kids gets out diapers we'll be ready to start building a workshop for the 21st Century.

For more fablab goodness, here's a weekend reading list:

Worldchanging embraces iFabricate
John Hagel sees the big picture
Fortune discovers DIY 

The dangers of "Headism"

I've been giving a series of Long Tail speeches recently, using Powerpoint as my muse. I actually quite like it as a creative medium (although not as much as some). Because it's primarily visual (or should be, when done right), it nicely compliments a blog, which is primarily textual. But speeches don't reach everyone, so this post is an effort to create a hybrid: blogpoint? Anyway, it's  mostly just graphics, with a bit of narration.

In these speeches I've been riffing on the dangers of "headism", the mistake of assuming that the economic incentives and other forces that dominate at the head of a demand curve apply equally down the tail. This is a common misconception, much like when people ask bloggers how they intend to make money from their online noodling (other than, you know, turning it into a sure-to-be bestselling book coming out RSN).

For too long we've viewed the economics of the entertainment industry through the lens of hits and stars, studios and networks. Just as we are recognizing that the Long Tail is a huge and growing market that was hidden by the scarcity economics of the old distribution systems, we're starting to realize the nature of the goods, the participants and the incentives in this new market are also different. Here are a few examples:

1) Once people learn to turn their head and shift their gaze from the far left of the curve, the hit domain we've been focused for most of the past century, to the niches at the right, they often go too far. Yes, the far end of the tail is an emerging area of grassroots production and liberated archives that will someday be a marketplace of millions. But in the near term, the greatest potential is in the middle, which is full of existing commercial goods that simply haven't found ways to tap all the latent demand that's out there.

Whether because of the tyranny of geography, the limitations of shelf space, the poor search tools of the physical world, or underdeveloped word-of-mouth networks to provide cheap marketing in their domain, most products  don't find the markets they deserve.  This category, which Jeff Bezos calls the "hard middle," is where most of the economic potential of the Long Tail is right now.

2) Likewise, the incentives for the producers and creators of these products change as you go from hits to niches; Madonna may be in it mostly for the money, but I sure wasn't when I slapped a bass badly in my misspent twenties. Most authors, meanwhile, write books to find readers, not riches (although those readers can lead to lucrative consulting fees, speeches or tenure; books are powerful marketing vehicles for personal brands). And plenty of up-and-coming independent filmmakers would be only to happy to have their movies freely spread far and wide on bittorrent to build their reputation.

And then there are the thousands of Garageband remixers, bloggers, Flash artists, game modders, and skateboard videographers who make stuff because they can. Powerful computers and software combined with the free distribution of the Internet is the greatest platform for personal expression the world has ever seen. Not everyone wants to be a star, and these days you don't have to be one to find an audience.

3) At the head, they've given up their day job. In the middle they're looking for a better day job  or building the one they've got. And the bottom it's not work at all (at least not yet).

Tweak this construction for various different media and industries, and you can see how the presumption that everyone in the marketplace is cut from the same cloth, just of varying quality, is a mistake. In a marketplace with limitless availability, the line between professionals and amateurs is a continuous spectrum, with lots of semipros in between.

4) Intellectual property is perhaps the domain most distorted by headism. Disney and the Dave Matthews Band may be doing all they can to embrace and extend copyright, but there are plenty of other (maybe more) artists and producers who see P2P distribution as free marketing, and PDF downloads as a way to increase impact and audience.

At the top, the studios, major labels and publishers defend their copyright fiercely. In the middle, the domain of independent labels and academic presses, it's a gray area. Some care about copyright and some don't. Yet the law doesn't distinguish between them, and thus their impact is limited by the scarcity effects of legal restrictions.

Further down the tail, more firmly in the non-commercial zone, an increasing number of content creators are choosing to explicitly give up some of their copyright protections using a Creative Commons license, for the sake of the greater value (for them) of free distribution, remixing, and other peer-to-peer propagation of their ideas, interests and fame. I've done it for this blog, for all the reasons above.

In one part of my professional life I'm near the head of the curve and in another I'm in the tail. My decisions on intellectual property are different in each. Someday soon, I hope, marketplaces and regulation will make it easier to reflect this reality.

(thanks to Andrew Blau of the Global Business Network for helping me think this through over coffee in Berkeley)

Is the Long Tail full of crap?

One of the most frequent mistakes people make about the Long Tail is to assume that things that don't sell well are "not as good" as things that do sell well. Or, to put it another way, they assume that the Long Tail is full of crap. After all, if that album/book/film/whatever were excellent, it would be a hit, right?

Well, in a word, no. Niches operate by different economics than the mainstream. And the reason for that helps explain why so much about Long Tail content is counterintuitive, especially when we're used to scarcity thinking.

First, let's get one thing straight: the Long Tail is indeed full of crap. But it's also full of works of refined brilliance and depth--and an awful lot in between. Exactly the same can be said of the Web itself. Ten years ago, people complained that there was a lot of junk on the Internet and, sure enough, any casual surf quickly confirmed that. Then along came search engines to help pull some signal from the noise and finally Google, tapping the wisdom of the crowd itself to turn a mass of incoherence into the closest thing to an oracle the world has ever seen.

On a store shelf or in any other limited means of distribution, the ratio of good to bad matters because it's a zero sum game. Space for one eliminates space for the other. Prominence for one obscures the other. If there are ten crappy toys for each good one in the aisle, you'll think poorly of the toy store and be discouraged from browsing. Likewise it's no fun to flip through bin after bin of CDs if you haven't heard of any of them.

But where you have unlimited shelf space, it's an infinite sum game. The billions of crappy web pages about whatever are not a problem in the way that billions of crappy CDs on the Tower Records shelves would be. Inventory is "non-rivalrous" and the ratio of good to bad is simply a signal-to-noise problem, solvable with information tools.

Which is to say it's not much of a problem at all. You just need better filters, such as recommendations and good search engines. The fact that screens 10 and beyond of your Google search results are unhelpful doesn't matters because screens 1-3 are so useful. The noise is still out there, but Google allows you to effectively ignore it. Filters rule!

The following is this expressed graphically. As you go down the Long Tail the signal-to-noise ratio gets worse. Thus the only way you can maintain a consistently good enough signal to find what you want is if your filters get increasingly powerful.

This leads to the key to what's different about Long Tails. They're not pre-filtered by the requirements of distribution bottlenecks and all that entails (editors, studio execs, A&R guys and Wal-Mart purchasing managers). As a result their components vary wildly in quality, just like everything else in the world.

One way to describe this (using the same language of information theory that brought us signal-to-noise ratios) would be to say that Long Tails have a "wide dynamic range" of quality: awful to great. By contrast, the average store shelf has a relatively narrow dynamic range of quality: mostly average to good (there's some really great stuff, but much of that is too expensive for the average retail shelf; niches exist at both ends of the quality spectrum).

So tails have a wide dynamic range and heads have a narrow dynamic range. Like this:

Note that you have high-quality goods in every part of the curve, from top to bottom. Yes, there are more low-quality goods in the tail and the average level of quality declines as you go down the curve. But with good filters averages don't matter. It's all about the diamonds, not the rough, and diamonds can be found anywhere.

Before I go on I should say a word about the confusing terms "high quality" and "low quality". They are, of course, entirely subjective. Here are some examples of criteria people might use to value content:

"High quality":

• Addresses my interests
• Well made
• Fresh
• Substantive
• Compelling

"Low quality":

• Not for me
• Badly made
• Stale
• Superficial
• Boring

Note that all of those are entirely in the eye of the beholder; there are no absolute measures of content quality. One person's "good" could easily be another's "bad"; indeed, it almost always is. I like economics papers, but I have friends who like Maxim. They find my stuff boring; I find their stuff superficial (except for those Jessica Alba pics). 

This is why niches are different. Your noise is my signal. If a producer intends something to be absolutely right for one audience it will by definition be wrong for another. The compromises necessary to make something appeal to everyone mean that it will almost certainly not appeal perfectly to anyone--that's why they call it the lowest common denominator.

Is South Park badly made or brilliant? Once upon a time some network exec had to answer that question for all of us before it could get distribution. Cable lowered the bar and now Netflix and the web are lowering it further. In a Long Tail world each of us answers the quality question for ourselves and the marketplace sorts it out.

All this leads to three counterintuitive lessons of the Long Tail.

1. Niche content can be of higher quality than hit content.
2. It doesn't matter how much junk there is around those gems; with good filters, the average level of quality is irrelevant.
3. You can charge more for high-quality niche content because it is so well-suited to its audience.  

Powerlaw Primer

I'm re-reading Clay Shirky's excellent early work on powerlaws and am reminded that many people are still confused about what exactly a powerlaw is and under what circumstances it arises.

The first answer is easy: it's basically any curve that has a y=1/x sort of shape, like the chart on the right. You can find powerlaws practically anywhere you look, from biology to book sales. The Long Tail is a powerlaw that isn't cruelly cut off by bottlenecks in distribution such as limited shelf space and available channels.

The second answer is easy, too. Powerlaws come about when you have three conditions: 

1. Variety 
2. Inequality 
3. Network effects (word of mouth, for example) to amplify the differences between them.   

In others words, powerlaw distributions occur where things are different, some are better than others, and network effects can work to promote the good and suppress the bad. This results in what Vilfredo Pareto called the predictable imbalance of markets, culture and society: success breeds success, rich get richer and so on. Needless to say, these forces describe a good fraction of the world around us.

The Variety Revolution

I find it reassuring that Virginia Postrel, the former Reason editor whose writings on culture and economics are a model of the form, is chronicling "The Variety Revolution", which is a close cousin of The Long Tail. She's just updated her website to include the category of  variety and choice, which currently includes her writings on these subjects going back to 1998. She describes it thus:

The "variety revolution" is one of the biggest business stories of the past decade. Thanks to production and distribution innovations, consumers now have access to far more choices for all kinds of goods and services, from fresh vegetables in the supermarket to DVDs from Netflix. I am exploring the management practices that have made the variety revolution possible, the psychological challenges it poses for consumers, and the opportunities it presents for both future business models and personal pleasure and meaning. The variety revolution is an economic story, but it has much broader implications for how we think about pluralism and individual differences.

USA Today, Longtailing bigtime

USA Today recently held a roundtable focusing on the Long Tail and published the highlights in a two-page spread last week.  It featured TiVo CEO Mike Ramsey, venture capitalist Roger McNamee, Firefox co-creator Blake Ross, and rapper Chuck D. Some excerpts follow:

Mike Ramsey:

What we've found is that the viewing patterns of people who watch live television — and are therefore restricted to prime time whenever they're home — are dramatically different than the viewing patterns of people who have the choice of just picking whatever they want.

Given the choice, people will migrate towards a much greater variety, and the deal is you've got to make everything available to everybody so that they're not restricted. And if you do, the market for that more esoteric, more specialized stuff is just as big as the market of the mainstream stuff.

Roger McNamee:

The people in the middle have tried to be arbiters of what we could be entertained by. They've been the determinants of what's a hit, what's not a hit. The great thing about the long tail is the consumers get to decide for themselves. They don't need somebody in the middle....What's been wrong is that capital, the money, has always been tied to distribution. The reason my firm exists is to change that, to put the capital with the content, with the creative people.

There's a start-up called Akimbo that's about to ship a product. Its initial programming will be soccer from Europe. It'll have things from India and from other cultures that have never been available because they don't have large-enough audiences to go on satellite or cable, but they have a plenty large-enough, and certainly devoted-enough, audience to go over the Internet.

Not only is it great to see industry leaders completely getting the Long Tail and putting it into practice, but it's especially gratifying to see it discussed with this level of sophistication in a publication as mainstream as USA Today. Credit to Kevin Maney, who moderated, for pulling together the right panel around the right subjects and asking the right questions. And extra special credit for the sidebar with a Long Tail definition that's spot on:

Last fall, Wired magazine identified a trend it dubbed "the long tail," and the term has since caught fire in tech and media circles. Basically, it says that in an era of almost limitless choice, many consumers will gravitate toward the most popular mass-market items, but just as many will move toward items that only a few, niche-market people want.

Take music in an age of Amazon.com and iTunes. A lot of music buyers want the hot new releases. But just as many buy music by lesser-known artists or older music — songs that record stores never would be able to carry but that can be offered online. All that small-market, niche music makes up the long tail.

Until the past few years, mass-market entertainment ruled the industry. In this new digital era, the long tail is becoming at least as powerful a force.

Personally, I think full-page treatment in both The Economist and USA Today in the same week should count as some sort of memetic hat trick, a test of full-spectrum cultural resonance. I can't decide which delights me more.

Eric Schmidt, Longtailer

At Google's first shareholder's meeting last week, Eric Schmidt, its CEO, elaborated on why he describes Google's mission as "serving the long tail."

He started by showing this slide:

Which he explained this way:

"We took a look at our market last year and asked ourselves: how are we doing? If you look at the advertiser, the market we're in, how do we do from the largest companies - Wal-Mart - in the world, all the way down to the smallest companies in the world, the single individual. We call this The Long Tail. A lot of people have been talking about it--it's a very interesting idea.

We looked at this and we said, we've been doing really well up until now in the middle part of this.  Well-run, mid-sized businesses, smart people solving interesting problems. But how well do we do against the problems of the very largest customers? So last year we brought out a whole suite of tools for very large advertisers who can use our services in all of their divisions to generate lots and lots of revenues because, of course, in our model the advertising drives predictability, it drives conversions, and so forth.

And what about the individual contributor, the small business, the company where Joe or Bob is the CEO, the CIO, the CFO and the worker and the support person--a one person company, a two-person company, a three-person company? We built a whole bunch of small, self-service tools which allowed them to almost automatically use this service.

So [we went] in both directions last year. By going all the way to the top, we were able to capture very large and historically underserved businesses as well as a whole new area that never had access to these kinds of online services."

For me, the main take-home from this is the importance of addressing both sides of the curve, a subject I'll elaborate on in the next post. 

Robot Child-Herders

We have a Roomba Discovery, which we love. It does an amazingly good job of cleaning the children's playroom, but perhaps not in exactly the way its inventors at iRobot intended.

Although it's fine at sucking up dirt, finding its way around the room and returning to its charging station, its real achievement is in not only getting the children to clean up their toys first but also tiring them out before bed.

This works with three magic phrases: 

1. "Roomba's coming out tonight. Clean up your toys or Roomba will eat them!" 
2. "If you can clean them up fast you can stay up to watch Roomba!" 
3. "Here goes Roomba. Don't let him touch you!"

The uttering of these three sentences results in the perfect end to an evening. The kids scurry around and pick up every last toy (it's the tiniest Lego pieces that get eaten the fastest), then race around the room jumping over Roomba as it drives from wall to wall, randomly changing direction just often enough to make the game fun. (We've told them that if Roomba runs into them it will think that they're a wall and not clean there, which may or may not be true.) Then, after 15 minutes of this, they're bored and ready for bed.

I love the thought that our children are growing up used to having domestic robots in the house. Robots for them are slightly dim but friendly vacuum cleaners, not fearsome weapons or fantasy toys. "Robot love me," declares the two-year-old.