July 28, 2004
Behold, the God Box
Wynn Quon
The hard disk drive is making miraculous progress. Call it Less's Law: The cost of storage is falling by half every 12 months, while capacity doubles.
July 28, 2004
Behold, the God Box
Wynn Quon
The hard disk drive is making miraculous progress. Call it Less's Law: The cost of storage is falling by half every 12 months, while capacity doubles.
Sometimes the biggest stories hide behind the smallest headlines. Yahoo! recently announced that it would be raising the storage limits on its free e-mail service. The curious part is how high the new limits are. Before the change, Yahoo! gave users 4 megabytes per account. Did Yahoo! double it to 8 megabytes? Nope. The new limit is 100 megabytes, an astounding 25-fold increase.
Yahoo!'s unprecedented giveaway is just one small example of the impact of a technological advance so rapid that it puts even Moore's Law to shame. It's a story of a fundamental force which has driven the growth of the Internet, the adoption of broadband and the ascent of killer apps such as Google. It's a story of unimagined new products to come as well as the death knell for old-school digital content providers like the music industry.
What we're talking about is the miraculous progress of a lowly device: The hard disk drive. While Intel's microprocessors have set a brisk pace by doubling in density every 18 months (following Gordon Moore's famous prediction), hard disks are on an even faster sprint. Hard disk storage density has been doubling every 12 months. Another pivotal fact is that cost has been plummeting by 50% a year. A megabyte of disk space cost US$1 in 1995. It now costs less than one-tenth of a penny. It's through this inexorable decline, megabytes costing less and less, that gives this technology its great disruptive power. While Moore's Law gets all the publicity, there isn't even a name for the blistering advance of hard disk technology. Why don't we call it Less' law? Less' law states that the cost of disk storage falls by half every twelve months, while capacity doubles.
Less' law has urgent implications for a wide range of industries. Here's some that come to mind:
Obviously, if it costs less to store data, the price of bulk data services such as hosting web pages, e-mail and data archival will fall. But there are also subtler side effects that ripple into other areas. Yahoo!'s raising of its e-mail storage limit for example instantly puts pressure on the makers of backup storage products like USB memory keys. With a 100Mb limit, an e-mail account can act as a free backup device. Want to save a copy of your PC files offsite? Mail it to your free Yahoo! account. Need more than 100Mb? Open up another account.
A corollary of Less' law is that the amount of stored data rises to fill the available space (we could call this Parkinson's commentary on Less' law). And the more data there is, the more data traffic. The bottom line: more digital content (both privately and publicly on the Internet)and greater demand for broadband connections at faster and faster bandwidths.
A third major implication is an increase in digital piracy. When Napster was on the air four years ago, the average hard disk size was 6 gigabytes. Bigger hard drives mean that the users of today's public file sharing networks like Kazaa and Shareaza can swap more files. The record companies, the industry hardest hit by digital bootlegging, has responded with lawsuits directed at heavy users of the networks. Less' law says that this approach is doomed. In Napster's time, a file-sharing network would need sixty users to supply the same swapping capacity that a single user has today. The result is that viable file-sharing networks can be launched with fewer and fewer members. Ten friends can easily muster over one terabyte of hard disk storage between them. Private file sharing networks will be the wave of the future. The music industry has had no success stopping Kazaa. Imagine the luck they'll have trying to stop millions of private no-name networks. Whether it's music, movies or software, Less' law enables piracy on an unprecedented scale. (In an incidental footnote to the piracy issue, Yahoo!'s new storage limits include an increase in the size of the attachments one can send to 10 megabytes. Since most songs are less than 5 megabytes in size, one can now use a public e-mail service to send MP3 songs to your friends.)
Look further down the road and Less' law demolishes the pricing model of pay-by-the-bit digital content providers. Let's take the music industry again as an example. A typical CD (encoded in MP3) takes 40 megabytes of space. Today's 120 gigabyte hard drive can hold up to 3,000 CDs. But consider what happens four years from now. After four iterations of Less' law, the typical $100 hard drive will be 2 terabytes in size (2000 gigabytes) and hold up to 60,000 CDs. It would be big enough to be a "God box", a single device that can hold the entire musical canon, everything from ABBA to Zappa, Albeniz to Wagner, acid-rock to zydeco. While the music business has historically sold music by individual tracks and by album, the emergence of God boxes combined with predatory piracy will force music providers to sell music in larger and larger volumes. Customers will be looking to buy entire genres of music, all the best folk music ever recorded say, or the entire EMI classical back catalog. And at progressively lower and lower prices.
Less' ultimatum to the digital content industries is that the price of a digital bit pattern tends towards zero. To stay alive in the world of God boxes, a radical change in focus is needed. A digital bit pattern can no longer be the main product offering, instead it becomes a loss leader in the sale of something else.
While Less' law makes life difficult for some, it opens up tremendous opportunity for others. We'll see new consumer electronic devices with enormous capacities. Digital video recorders that can hold months of television programming. A movie jukebox that holds 250 top movies. A personal recorder that could store your life in digital format - all your e-mail, your documents, your photos, your phone calls, everything.
Another opportunity lies in helping people deal with this tsunami of data. Google is an early example of success in this area. Google has benefited firsthand from Less' law. Plunging storage costs allowed Google to grow its massive server farms at lower and lower cost even as the Internet it searches through grows ever larger. (It's no surprise that Google's next big splash shows Less' law hard at work. This year, Google will launch G-mail, a free e-mail service. How much storage will each Gmail user have? 1 gigabyte, ten times Yahoo!'s already bountiful limit.) Google's search engine is the first-generation in a whole new class of products that help crystallize gems of useful information from mountains of raw data. We'll be needing more of this type of help. A lot more. We'll need tools to sort efficiently through our e-mail files, tools to search image and video data. We'll need "data scrubbing" services to certify that our databases are up-to-date and correct. Businesses will want to mine their customer databases to get insight into buying patterns. The music lover with a God box will desperately need to organize the songs in a meaningful way. Having 60,000 CDs is nice. Having a smart guide to help you through this musical cornucopia would be essential. Do it in an entertaining fashion and it'll be a killer app.
And the dark side of Less' law? Governments will find it easier to accumulate massive amounts of private personal information. Privacy advocates are unlikely to be enthralled with new video recorders that can capture months of surveillance. But new worries also enter the picture. Consider for example that the income tax returns of all Canadians will soon fit onto a single pocket-sized hard drive. While data security has focussed on online hackers, the era of small terabyte drives means that someone could physically walk away with the contents of such highly sensitive databases. Indeed, the combination of cheap storage and peer-to-peer networks creates a whole new category of data crime: the deliberate theft and subsequent publication of sensitive databases on a public file-sharing network like Kazaa. To some the "liberation" of the nation's private tax returns would be the ultimate cyber-political stunt. To others (like those who create malicious computer viruses), it would be worth doing for the havoc alone.
Despite the drawbacks, there's no question that Less' Law has been on the whole a positive technological force. Together with the exponential growth in microprocessor power, Less' Law fulfils technology's perennial promise: Doing Moore with Less.
Wynn Quon is Chief Technology Analyst at Legado Associates( www.legadoassociates.com) He can be reached at wynn_quon@hotmail.com.
