Super Size Storage - (He Just Can't Get Enough)

[Jonathon Watkins]

<div class='os_post_top_link'><a href='http://news.bbc.co.uk/1/hi/magazine/3673262.stm' target='_blank'>http://news.bbc.co.uk/1/hi/magazine/3673262.stm</a><br /><br /></div><i>"Now 32 petabytes - that really is a serious amount of storage. A 32 "pet" media player could hold about seven billion MP3s or roughly 50 million movies - surely that would be enough for even the most avid cinema buff or music nut? If I've learned anything from Depeche Mode and my old 64Mb player, it's that 32 petabytes probably won't be enough. So it must only be a matter of time before the exabyte, zettabyte, yottabyte and finally the brontobyte media player appears on the market. A brontobyte is million million petabytes, enough to store everything that's ever been filmed, taped, photographed, recorded, written, spoken, and probably even thought. Still, it would probably be wise to wait for the 10 bront model, just to be on the safe side."</i><br /><br />The BBC has just put up an amusing, but thought provoking article about digital storage space. The columnist tells a tale that most of us will recognise. When he started out with his 64Mb MP3 player he was happy with the capacity, and now the sky is the limit. Sound familiar? He talks about how the current generation 40Gb MP3 players should be large enough for <b>most</b> folk's music collection and speculates about the emerging generation of 'personal media' players and their insatiable demand for storage capacity to fit in multiple films. Maybe it's just me, but I don't have a burning desire to carry around 50 films me all the time. If you do a lot of long distance travelling I can see how that would be useful, but how many folks will want regular mobile access to that that number of films? <br /><br />Do you reckon that the BBC columnist is correct in that demand of mobile personal storage will inexorably continue to rise? I have 35Gb+ of music and 30Gb+ of data (though that's increasing fast with an 8Mp camera to add more photos). As I don't make digital videos or collect films, I just don't see my storage needs making a step change. Sure, it adds up, but have you seen the prices of 250Gb hard disks recently? Heck, at those prices I'll have two and do RAID 1 disk mirroring for peace of mind. Are potential disk capacities rising faster than our abilities to fill them, or are we chasing rainbows, never able to connect? At what point does the increase in storage space catch up with our wants, let alone our needs? Thoughts?

[Shuushin]

My opinion is that the portable storage will be the focus; how much can we bring with us. Its more of a "peace of mind" thing. Who really needs to have 40Gb of music and and ultimately a movie collection in their pocket? (W?BIC - this is from guy who runs all that crap in his car from a PPC).

Having said that, wireless (or even wired) access to the media will certainly be a factor to potentially offset the need for on-line storage capacity.

"Potentially", because I suspect, since no one is in charge and that these are basically two competing technologies (telecom and on-line storage), that both will increase in capacity to rediculous proportions while the costs decrease.

It's like the first time I got a 10 gig drive for my PC - I started a campaign to snake all the internet porn I could find, "just in case". After realizing that it was all gonna stay there anyway, and that there is slightly more than 10 gig of porn in the world... I came to accept that I really didn't need that much space. Of course, now there is music - but it is eventually going to lead to the same conclusions.

Anyway - we're always gonna want more. More and Faster.

[JvanEkris]

I've thought about this a great deal. On my Multimedia-server i have about 250Gb of movies in high-quality AVI. Reducing the size to PocketPC-format, it would become about 60 Gb. Add a 8 Gb of MP3's, and i have all my movies and music with me. Add another 10 Gb and i have all my personal photo's with me.

Why? Because i do not want to be bothered with the notion that i have to decide in advance what i want to see/hear. When i'm traveling i want to decide at THAT moment what i want to see. That could be radically different from what i anticipated that morning. When i had a rough meeting, i want to see blood splashing on my screen. When i had a relaxing day, i want a good thriller to tease my brain. But i can not predict what my mood i will be when i want to see a movie. I want to decide when i'm on the move what i'm going to see, and if the movie bores me halfway i want to take another movie.

Having to decide what to take with me is ridiculous with that in mind. I want all my music with me and all my movies with me to decide what i'm going to listen/see on the move. IMHO, synchronizing music every day is a worthless excersise in order to keep people busy......

Jaap

[DinarSoft]

And that's not all,
New compressed-media formats can also increase the amount of media/data files you can carry with you.

:jawdrop:

[Felix Torres]

Right conclusion, wrong rationale. ;-)
Big cheap portable drives will be desirable, not just for video, but for all kinds of media; most especially music and photos.

It is worth remembering that the current "standards" of compressed data, Mp3, WMA, AAC, and JPEG are all *lossy* formats. With a 100Gb in your digital jukebox you could carry 100-200 cds in a lossless format instead of the lossy formats now in use. With video, you'd be able to store the video at ED or better resolutions instead of QVGA. Same number of files, higher quality.
The same goes for images; the recent Adobe digital negative format is geared in that very direction because massive portable storage is becoming so cheap that soon, lossy formats won't make much sense for many applications. Certainly in digital photography, the availability of 1GB flash cards for under $100 makes JPEG less attractive than RAW since a photography session is not likely going to require 500 photos...

Nature abhors an empty hard drive, and one of the most common ways to use storage space up is by transitioning to higher quality data-files. ;-)
Which we will.

[that_kid]

Right now I have a 40 gig iPod and that only holds about 30% of my music collection. In another six months after I have converted more of my vinly over that percentage will drop. I hate when I have to decide on what should go on my iPod, in the car I have a neo car jukebox with a 200 gig drive in it. Enough to hold my music collection at the moment but it's getting smaller everytime I add something new.
When I first got into computers a 1.2 mb floppy was considered big enough for just about anything, when I started college our class was the first class to receive computers with hard drives (40 megs). Three months after getting those drives we were doublestacking them. You may not need the storage space today, but you will need it as you go. With the advancements in portable media players now people will want to carry as much with them whenever they can. Reminds me of my high school days carrying all those cassette tapes to listen to in my walkman. For me it's all about the data. I'm even making plans for a SAN at home.

[jgrnt1]

The average person (read "not us") will not take the time to rip everything to whatever storage media/device is available. However, I can see the day coming when hard copies of music, movies, etc., will be replaced by files sent directly to the home. It will take more bandwidth, huge storage capabilities (think about everyone's DVD collections in high-def on a hard drive) and will almost surely come with half a dozen, incompatible, Draconian DRM schemes.

Storage capacity, processor speed, graphics speed, etc., usually far exceed the needs of the average user. We don't see that, because we find uses for all the speed and storage. This stuff will become ubiquitous when there is a reason for the average user to have it.

[bjornkeizers]

My opinion is that the portable storage will be the focus; how much can we bring with us. Its more of a "peace of mind" thing. Who really needs to have 40Gb of music and and ultimately a movie collection in their pocket?

I wouldn't go so far as to say we 'need' it - but life sure sucks without it! I bought an Archos Gmini 400 20 gig multimedia player earlier this month. I don't have that many MP3's, so I started ripping my own movies. I can store around 40 movies on it; since I rip them to a lower resolution. 40 movies.

40 *full length* *good quality* *movies*. And that's with just 20 gigs. Imagine a 200 gig drive: 400 movies. A 1.000 gig drive: 2000 movies.... and so on and so forth. If all these scifi like specs do come true, I'll be able to carry every movie ever made on a postage stamp sized card by the end of next decade....

[manywhere]

Ahh... this article reminds me to start building my super distributed file system cluster based Beowolf clusters. ;)

[svenllr]

I have over 75GB of MP3's and M4P's and it grows weekly. So, 40GB just isn't enough for me. 64MB wasn't either when the first player came out hence my holding off the purchase of my first MP3 player until the iPod 30GB.

One thing I've learned through my computing years is you can never say "I have enough". With progressing technologies, demands just get bigger. MS and their quest to be the next HD DVD standard will trickle down to portable players and those files won't be small. Music will follow suit. SACD and DVD-A like formats will be here soon and they will be large files and we'll need those petabytes, trust me.

[JvanEkris]

The average person (read "not us") will not take the time to rip everything to whatever storage media/device is available. However, I can see the day coming when hard copies of music, movies, etc., will be replaced by files sent directly to the home.Do not underestimate John Doe. I always thought that both my neigbours were completely computer illiterates until one day i discovered two WiFi Access points besides my own, properly configured with WEP encryption etc..

Most kids now enough about computers to just download the music and movies. They do not seem to care about the legal/moral issues involved (but that is another discussion). However, these kids do not need to rip anything anymore, they alread have it in encoded format on their harddrives........

Jaap

[jgrnt1]

Do not underestimate John Doe. I always thought that both my neigbours were completely computer illiterates until one day i discovered two WiFi Access points besides my own, properly configured with WEP encryption etc.

It's quite possible that you have some computer literate neighbors. It is also possible that they are clueless and paid somebody to install it for them. I just fixed a neighbor's PC last week. He's set up with DSL, WiFi, his wife has a laptop, properly configured (WEP, etc.). They haven't got a clue how any of it works, or how to fix, tweak, update, etc. Their DSL provider installed it for them. Their PC was so full of spyware/adware that it was choking to death.

I think the old joke about the VCR blinking 12:00....12:00....12:00 still applies to many people. Once it's idiotproof and inexpensive, it will become commonplace, but I think we're still a ways away from that.

[mikewoodhouse]

I've just started uploading to my new 40GB IRiver and I'm trying to muster enthusiasm to move archive video files from my PC to my new Maxtor 250GB external drive. Here I was thinking I'd was getting ahead of the game and people are already looking forward to petabytes. :roll:

I need to lie down.

:confused totally:

[danmanmayer]

Most people are starting to get higher quality audio files better than 128 kbs. Many are getting into the video stuff. Digital camera's take up more and more. Being a video editting guy for a hobby i know how fast video adds up and most of that is sitll pretty low quality video. Start working with HDTV quality and it is oging to take out 250 GB drives fast. So being a home user that has 600 GB of hard drive space (FULL!) I dont think any amount of storage space is going to be enough because i will always find some new and amazing applicationg media format or something else to fill it.

So bring on the broncobyte... hehe

[Darius Wey]

Now this is the kind of topic that made me wonder the other day:

If you were to have the MP3 of every recognised song that ever existed (including remixes) at 192kbps, how much space would you need to store it all? ;)

Suddenly, all these cool *-byte terms sound nice. :D It's also nice on the wallet too. :P

[picard]

If you were to have the MP3 of every recognised song that ever existed (including remixes) at 192kbps, how much space would you need to store it all? ;)
let's see the other way around (from a personal music player viewpoint): you listen music for 80years non-stop (even in sleep :) ) with 192kpbs = ~55TByte

[JvanEkris]

But that does not include my very triggerhappy "skip" finger.

Jaap

[Jonathon Watkins]

Ahh... this article reminds me to start building my super distributed file system cluster based Beowolf clusters. ;)

:lol: Very good. MORE STORAGE PLEASE methinks! :wink:

[Jonathan1]

My .000000002 cents.
The thing is at some point we are going to hit a storage capacity where you can easily store every movies, song, or other media ever created in pristine, lossless, condition that can’t be distinguished from the original source material and still have enough space left over to download every piece of porn off the net. 8O ;) At which point there is no real need to go above that storage capacity.
If you compare today’s drives to yesteryear’s you will see amazing growth in a relative short period of time, but as drive size has increased apps that take advantage of such drives are few and far between. Sure they exist and it’s easy to fill up even a 250GB hard drive with multimedia files but I’m talking the average person and the average desktop. A person who rips their entire DVD collection isn’t exactly average. As time goes by the capabilities of these drives are going to further outpace the ability of the typical user to fill up a drive. In 10 years the average and even the relative above average user won’t fill up their hard drive for the life of their system. Ob course you will always have the uber geek to be around to fill up that 1PB drive and ask for more but we are few and far between.
Where such huge drives WILL come into play is NEW media/data types that haven’t been invented yet. Virtual 3D worlds for video games or location fly-arounds for MS Encarta 2018 stored in high definition so good it will put reality to shame. Or possibly 3D televisions with TIVO 20TB or something. Some form of data that is going to be drastically more storage intensive then anything we have right now. (Think high resolution scans of your body via nanites or some sci-fi futuristic stuff.)
Also I think at some point the notion of compression is going to become totally obsolete for anything other then data transmission.

*shrugs* just my two cents

[Jonathon Watkins]

It is worth remembering that the current "standards" of compressed data, Mp3, WMA, AAC, and JPEG are all *lossy* formats. With a 100Gb in your digital jukebox you could carry 100-200 cds in a lossless format instead of the lossy formats now in use. With video, you'd be able to store the video at ED or better resolutions instead of QVGA. Same number of files, higher quality.

Well, I can just hear the difference between 128 and 160 kbp MP3, but I can't hear the difference between 160 and 192 kbp MP3 files, which is why I rip at 192, to ad a bit of margin. Playing lossless tracks would not give me anything 'extra' I could appreciate. I can obviously see the value of going to native resolution for video, but again, once you hit the 'optimum' resolution, there's not anywhere to go. (Until HDTV I suppose :wink: ).


The same goes for images; the recent Adobe digital negative format is geared in that very direction because massive portable storage is becoming so cheap that soon, lossy formats won't make much sense for many applications. Certainly in digital photography, the availability of 1GB flash cards for under $100 makes JPEG less attractive than RAW since a photography session is not likely going to require 500 photos...

Well,,,,,,, it depends on your photographic style and what you are photographing. :wink: I have gone through 700+ photos at an airshow and took 1300 at this year's Greenbelt festival in the UK..... Still, I am looking forward to gettning a standard for a 'digital negative'.

[quote=Felix Torres]Nature abhors an empty hard drive, and one of the most common ways to use storage space up is by transitioning to higher quality data-files. ;-)
Which we will.

Good, fair points. So, you conclusion is, that the rainbow will be forever out of our reach? More, more, more for ever and ever?

[LarDude]

Forget about it. Our "standards", what we want to see/do/store, will always rise to "meet" whatever capacity vendors/technology can supply.

We'll come up with newer/"better"(i.e. more-memory-intensive) ways of doing the same thing plus a few new things which we haven't even thought of yet. Who knows!? Adding a hi-res "always-on" camera (don't ask me how it would be powered) to an iPod/MP3 player/PDA/cellphone with "petabytes" of memory and, who knows, it could become our very own personal "short term memory augmentation unit", recording everything we do, where we've been. "What did I have for lunch 2 days ago?" you ask? "No problem. Let me scroll thru my PSMA (Personal Short-term Memory Assistant) and I can show you what I ate, how long I took to eat, and how many gravy stains I added to my tie".

Remember the "genius" who said:
"640 kB of RAM should be enough for anybody!"

[Jonathon Watkins]

Remember the "genius" who said:
"640 kB of RAM should be enough for anybody!"

An urban legend. He never said it..... (http://tafkac.org/celebrities/bill.gates/gates_memory.html)

[LarDude]

Remember the "genius" who said:
"640 kB of RAM should be enough for anybody!"

An urban legend. He never said it..... (http://tafkac.org/celebrities/bill.gates/gates_memory.html)

Oh, OK. Sure. Well, ahem, what he claims must be true then. :mrgreen:

[bjornkeizers]

And the good part is, most of us will still be around when we hit those magical limits. *knocks on wood* I can hardly wait.

[JvanEkris]

A person who rips their entire DVD collection isn’t exactly average. As time goes by the capabilities of these drives are going to further outpace the ability of the typical user to fill up a drive. In 10 years the average and even the relative above average user won’t fill up their hard drive for the life of their system. Ob course you will always have the uber geek to be around to fill up that 1PB drive and ask for more but we are few and far between.Following your argument, you can not explain the current huge sellings of the 20Gb iPods. Only thing it can play is MP3. THat is a huge amount of music! I use MP3 at 192Kbs since 1997, swapped with a dozen of people, but even i can not fil 20Gb of MP3 music! Still the 20Gb iPod is a huge success. Apperantly people have a need to take every song they have with them. Apperantly they do have the time/expertise to rip/download these huge numbers of music to fill 20 Gb.

Therefor, I have to disagree with you. People just start downloading movies, they are already doing it and i think it will become bigger than it is. Do not underestimate Joe Average and especially his kids who use smart tools to download everything they can get their hands on. Especially when they can get things for free, people are very keen on doing some tricks to get their hands on something.

Here in europe internet-providers even advertise with the fact that you can download movies in the matter of hours. This does miracles for what people do with their broadband connections (about 80% of all internet-connections now i guess). What i understand from analysts of webtraffic at internet-providers, people are downloading about anything. Internetproviders increase the downloadrates almost on a six-month basis, so it will become faster and easier and therefor more accessible.

Philips and others just have introduced new players (Tivo-like) that rip the DVD for you in a very easy manner. So borrowing a DVD from a friend and ripping it to your own drive will become common practice as well. It also records TV in a fashionable way to your harddrive, so basically it does everything you want to digitize all your video. This makes the available base of ripped video bigger and bigger.

And i think that will result in huge drives with data, that eventually people want to take with them when the are on holliday etc. just to amuse them when it is raining, amuse the kids when your driving, amuse yourself when you are on the plane or waiting for a train.....

Jaap

[JvanEkris]

And the good part is, most of us will still be around when we hit those magical limits. *knocks on wood* I can hardly wait.I hope so as well, but i think we will start acting in a different way that makes it easier to fill as well.

You see this happening in digital photography. In photography you had the magic 36 pictures on a film. If your picture was not perfect and you kept on messing up, you ran out of expensive film. The result was that people were very concious with the pictures they did take. This resulted in very low volumes being shot. I was considered very abnormal that i shot about 500 pictures on a holliday in Rome. Nowadays you have a couple of SD/CF cards and people start thinking: "it costs nothing and if it is no good i will throw it away aferwards". The result is that average people have a 1Gb SD card in their camara and another one in the bag and fill them in a weekend. Because they are affraid to miss anything and have the feeling they can clean it afterwards. They don't, so they have 2 Gb of pictures.....

I see a similar behaviour on my laptop (80Gb Harddisk). I have all the space i need. I write a lot of reports, but 10 Gb is the maximum needed for them. The OS takes about 5Gb. Leaves me 65Gb to put music on it. My collegues have about the same size of disks. We are swapping gigabytes of music through our company network. Some artists i never heard of, but i will give them a try and why not download all his/hers repertoire from my collegues drive to get a good impression. Why, because it does not matter anyway, and it fills up a very empty drive.

Now to movies. First of all, at the present we all are very neat when ripping DVD's (see Jason's reference to ripping DVD's, how important it is to set the bitrate...). With luck you can put about 16 movies on 4Gb, which is already a pricy thing. Right compression ratio, right screen-size etc.. I mean, when i put video on my pocketpc i will transcode the movie to 320x420 first and drop the rate to about 200Kbs, before i put it there. But when i have more space i will probably start to skip this step. Having this space also let us improve the sound. Now we are all satisfied with MP3 stereo, but why not use high-quality AAC 5.1 surround? THis will results in even bigger files.

When i have huge disks, i and a lot of people with me wil become more sloppy in encoding and start swapping movies with collegues, just to see what they like. This will result in huge data-storage needs.......

Jaap

[DinarSoft]

And that's not all,
New compressed-media formats can also increase the amount of media/data files you can carry with you.

:jawdrop:

[Darius Wey]

let's see the other way around (from a personal music player viewpoint): you listen music for 80years non-stop (even in sleep :) ) with 192kpbs = ~55TByte

80 years non-stop and sleep included? :O

I'd think I'd go insane first! >_&lt;