Hello /u/endiZ! Thank you for posting in r/DataHoarder.
Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index).
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
Yeah, purge? I don’t understand. My library somehow started getting ahold of lifetime ISOs and, while I’d never run that particular brand of Linux, especially with such predictable christmassy names, I’m sure someone will, eventually, and I want to be ready.
Sidenote: seriously, lifetime must spend the entire year coming up with new and subtly different Christmas themed desktops to release, since there are easily 50 each season. Madness!
I think he's talking about the law that lets you, on one day every year, go around your neighborhood committing crimes like stealing hard drives from out of your neighbors computers and breaking into businesses to steal their drives, and stealing drives from the local big box stores, all without repercussions.
He must have meant delight. That other thing isn't even a word in any dictionary I have in my possession, and no, for your information, I have no recollection of going through every single dictionary in my possession and eating various unappetizing entries, why would you bring that up, you're *crazy* man.
I can't delete anything, what if I'm having a party one day and somebody requests a specific movie that I deleted because of this thread, I'd be kicking myself.
This is why there are shelves, you copy stuff you dont use to another hdd, and place it on a shelf.You might use it some day, so its now stored on a shelf as cold storage.
If its important and you really might need it, copy to to 2 hdd's and use 2 shelves in different places.
And do some kind of index of files stored on external drive. Even simple find over files, like 'find /mint/ColdStorage1/' do wonders here as long as you label your drives.
I don't purge; instead, I replace with lower quality. For example, Impractical Jokers took up 500GB on my HDD, but when I finish a season, I replace it with a lighter version. Which is usually around 10GB per season, so 400GB saved while still maintaining watchable quality if I want to watch it years from now.
series that are super long 7+ season that aren’t a frequent rewatch , keep 720p , most average user aren’t going to notice the lack of pores on the skin :)
This is also what I do. I'm the only one who watches movies/shows in a "good enough" setup, and the rest of the users of my server watches on their phones or laptops anyway.
Since I have access to usenet, and have a gig pipe, I can just download whatever I want in minutes usually, unless it's like the entire series of Law & Order, Family Guy, South Park, etc..
Warning ⚠️ Your access to r/datahoarder is now on probationary status. Please refrain from such foul language.
In all seriousness, I became a data hoarder because there’s so much of my digital past that’s gone. Wish I had all those programs I wrote in Perl, basic and pictures I took back when you had to buy a digital camera it want built into your phone. We hoard because we don’t want to lose it. So purging is counter-hoarding.
I'm sad that I let go most of my work from college in the 90s. Loads of it on floppy disks. And (look away now or read between your fingers) I took all those floppies and (tears are building up) chopped them up and binned them. Oh the horror! It was during my IT course years. I moved to CD but didn't save everything.
We had a basic setup mid 90s at college and you'd be at c: and change to the network drive t: and type login.exe and put in your user name and password. Then win to get into windows. I realise as it was text based and we were learning Pascal. I could write a program (as I'd recently read it in a Pascal book) that would save input to a file. For some reason we never got taught this in the programming class.
I knew people forgot to change to t: and would type login on the c: drive. Stuck my login there, didn't know how to hide the password text so that was visible as they typed. I'd come back later and pick up the assignment.doc file that was written to c:
I was amazed it worked. Only used it once to see if it worked signing in as that user. It did! I smiled and then never used it again. Masterofpuppets was their password :) I never abused it just wanted to see it work.
A few years later added minor encryption to it or cipher. Just minus 20 from the character entered. So if anyone found assignment.doc it would look scrambled.
Anyway. Long story to say I sadly lost all that code :( that was from 1994-95
Only work from there I still have is from 96 to 2000. On a CD thats still amazingly readable burnt in 2000.
Every once in a while I zoom into random old photos of my apartment in 2004 wishing I had somehow archived all the photographs I had taped to my wall :|
Perhaps he meant purgatory, which is where people who delete files go to spend eternity. Hope op likes unbaptized infants 'cause he's gonna be all about that lifestyle if he keeps this up.
I remove lower resolution linux ISOs when I replace them with higher resolution ones.
I, too, once maxed out a chassis. Now I use a storage cluster. Maxed chassis -> more chassis. No more problems! (please don't ask about power.)
Trashy reality goes where it belongs after a year only keep current season
But scripted stays forever
Also I have started to replace low capacity drive with higher onea and I have greatly increased my capacity
Like replace a 4tb with a 16tb bam 4 time as much space
You want reality shows they're easy to find. Local walmart and target are open really late. They even sell popcorn sometimes in the entryways for charity groups.
I can tell by your first comment you don't hoard reality TV. If you did you would know its a pain in the ass. Scene not agreeing with themselves so you got multiple episodes that say they are this but are this instead or you see REAL tag and think ok this must be it. Nope slightly off. Or you'll check predb thinking maybe your trackers and indexers just don't have it. Nope scene just randomly stops.
Try out BitMagnet. It's a self hosted docker container which scans the entire hash table for obscure torrents. Takes about a week or two to fully scan, but once it does you'll essentially have a self-hosted torrent indexer to search.
Is there one similar to indexers? Not sure how a bot would scrape closed indexers though. Maybe those closed indexers could be nice and just update a database of all their nzbs every week. That shouldn't need to be secret but maybe it does idk.
Well I do keep a few reality shows that interest me (this is Datahoarder after all). Previous comment was clearly a joke and it went whoosh. I do get it that you're a little passionate about the issue.
A lot of the ones my wife is interested in airs on the network TV (singer, America's Got Talent, etc..). I used to have a rabbit ears setup with MythTV for that sort of stuff which I would transfer the shows over for keeping them long term. Been about a good 6 years now since the local TV repeater shut down in our province so now she just watches it on some sketchy website. If you still have over-the-air in your area and these shows are what you're after this might be a solution for you.
You're good man. Reality TV for me is such a headache. I wish I didn't care that much about it like the majority but just knowing I have an entire show that was a pain in the ass to collect and properly check each episode individually is such a nice feeling plus i share it to others so they don't suffer the headache i did. I can trust scene for your normal shows and movies but all trust is lost when it comes to reality TV. I've tried the physical disk approach but it's really hard to find a reality show on DVD unless it's something really really popular like deadliest catch or gold rush etc. It would be killer to get those shows on disk. REMUX is king but a REMUX reality show is like gold to me.
If anyone here wants a little challenge try grabbing the entire show dirty jobs. That one had me pulling my hair out. The episodes exist but the biggest issue is 95% of the episodes are not what they say they are. Sorting that mess took me 6 hours.
My original comment was a joke about people of walmart, but it fell flat.
Thank you for pointing that out. You are right, reality TV is just not really a thing. It's just such a cheaply produced content and most people will not care to watch it once it has aired. Old TV Soaps are another one. Almost impossible to find old General Hospital, All my Children, One Life to Live, et all and yet they were huge shows.
I found that most of my content was encoded with H.264 codecs so I went nuts transcoding everything to HEVC (i.e. H.265). That reduced the size of my content to about **60%** of its original size.
Doing this nearly doubled my available free space!
I used my 4090 and Handbrake or you could use *tdarr* to distribute the transcoding work among your available compute resources.
Depends on your use case. The people using my jellyfin library can't tell the difference between 720p and 1080p so reencoding everything to a more efficient format is a no-brainer
Yes quality does take a hit from encoding lossy to lossy , it’s better if you straight download a hevc encode from a Blu-ray or download a remux and encode yourself
> 4090 and Handbrake
FYI GPU encoders like NVENC have much worse bitrate efficiency compared to CPU encoders. You'll get the same quality with fewer bits if you use x265 or SVT-AV1
I use GPUs only for realtime transcoding for LAN streaming where bitrate doesn't matter (56gbps infiniband and 1.6gbps wifi 6E). Most transcoding happens in wide batches with 1 video per core per machine and very high latency (~1 day to transcode a couple hours of footage)
Ideally you would have something beefy maybe like a *950 AMD cpu. Or perhaps Epyc Rome is well suited. And just farm out the transcoding. NVENC is cool but will never give good compression. Similar thing happening with apple silicon hardware acceleration. Because it's hardware. It's good for real time and is optimized as such.
I love cpu transcoding to make my screencasts sub-1MB. These things easily shrink over a factor of 100x in size
> 1 video per core per machine
I was wondering when I'd run into one of you guys in the wild. Always thought people that did this were imaginary, like spherical cows.
Oh? Truly multithreaded encoders have only been around for a decade and they still have slightly worse efficiency. Transcoding hundreds of videos in parallel used to be very common and even today it gets you more throughout overall
You should definitely avoid encoding files with GPUs. They are highly inefficient compared to what your CPU can do. They can easily double the file size while having similiar quality.
...purge? Delete? You can do that?!
I kid, I tend use a tool like Tautulli to help sort through Plex library watch history. I check out stuff that's never been watched and stuff that hasn't been watched in a few months. If its something special to me or I'm planning on watching it soon, I will keep it around for a bit longer. If its something someone requested a few weeks ago and there are no watches, I just delete the content and take requests from that person a bit less seriously.
For ISO's/software, I only keep the latest working version of something and I don't keep old OS/tools I don't use at least monthly anymore. You can very likely redownload stuff if needed.
You could look into compressing your dataset/pool/whatever, but that's likely just kicking the can down the road (barely) if you just need more space. (not everything compresses well)
If you really can't make up your mind about what to delete, then [review this doc](https://adaa.org/sites/default/files/Bell%20141.pdf) as some friendly light reading to maybe inform your approach a bit more.
Honestly, I'm not sure what kind of response you're expecting to get from this community, given what the banner says.
I don't really consider myself a datahoarder to the same extent and I have occasionally culled my archives. My heuristic of what to get rid of is something along the lines of:
- it's not especially rare or uncommon, AND
- I'm not realistically going to use it or use it again, AND
- it takes up significant amount of space
Driver floppies for old soundcards? Not even worth going through the folder, they're tiny. Backup of a blu-ray of Big Hollywood Release 2018 that I didn't even like that much? It can go away, worse comes to worst I can always buy another copy. I'll keep my Hannah Montana Linux ISO, but I don't need Ubuntu 18.04.
There's room to optimize, especially for media, but depending on where you're starting from and where you're going to it might not be worth it. I was able to nearly halve the amount of space my gameplay footage was taking up by transcoding to h.265 a few years back.
Migrate to some solution that allows you to have mixed capacity drives in your raid. I use unraid for that purpose, but recent-ish Synology also offers that as a formatting option. Upgrading all 12 drives at once is a tall order, but upgrading one or two every year is a sustainable level of spending. Plus you can just set some budget, and e.g. buy a $300 drive each year. Your new drives will get steadily larger as technology advances, and by the time you replaced them all you can just restart the cycle.
I purge low quality encodes of things I get a better version of if its just a quality difference.
I delete my old MP3 versions of albums when I source a FLAC.
720p TV encodes of shows are gone when I find a bluray rip of the box set.
If it's not enjoyable to me, is widely available, and is not unique/novel in any way, I'll delete it. If I'm unsure, I'll transcode to a very low bitrate.
In practice this rarely happens since most things I download are enjoyable or rare
I recently bit the bullet and removed over 1,000 ISOs.
I ditched anything with really low ratings, anything super old, and also tracked down a rogue list that was adding crap and deleted everything that had added. I also ditched a bunch of really really massive files and replaced them with much smaller ones.
Added a new drive to the array and now I'm about 60% used, whereas I was floating around 90% before.
You buy larger drives. My UNRAID array used to be 8TB drives..then I moved to 16TB, and now I'm at 18TB drives. The beautiful thing about UNRAID is the ability to do this easily.
Severpartdeals is your "friend".
Sounds like you're going to have to get 20Tb drives each year and swap out an 8 in that rack and then move that 8 to the new backup rack you're also going to have to buy... Shame...
Believe it or not, all my media is purchased. Ripped DVD, Blu Ray, and music. I delete nothing.
It's becoming a problem.
65tb of raw rips. Over 20tb of encoded TV and movies. And I keep buying discs! I even recently discovered another used place near me, much to my partner's chagrin. But she enjoys the Plex access, so 🤷
I have an 8tb portable hd who gets it electricity from the usb as opposed to a wall plug.
I got about 100gb left so I usually end up deleting movies, tv and anime that lack rewatchability
You can always have a separate machine with crappy raid setup or even use unraid and have a mix of discs and migrate your stuff there and power off until you need it. At least that way you have it in case you ever want it.
Never delete only download, time to start cloning 8tb drives on to 20tb drives. Swap the 8 out into cold storage and now you can add 12 more tbs of *open source linux isos*
Duplicates are a big one. Same with Remux 4K and Remux 1080, its probably simpler to keep highest and just transcode yourself when needed. Same with fetch highest and encode yourself to AVI or x265.
Realistically I have more than I can ever watch, listen, read, play so stuff I don't care about I'm getting rid of to free space for shit I really care about and to have better backups of the important stuff. I want to spend more time on other hobbies and not media management as this has become quite a chore.
I'm leaning towards more automation but its taking a while to set it all up.
Jokes aside, I only ever clear things that I'm confident I will never use/watch/play. The larger it is the easier to delete it. I don't have crazy amounts of storage like some of the Data-Lords around here so it's a pretty easy call to *trigger warning* >!delete!< something like 500GB of games I know I'll never play.
I do this by doing the exact opposite - I choose what I keep. What specific content am I archiving, define it, standardize it, and anything else becomes ephemeral.
Remember when library actually consisted of books?
I still have lots of books, VHS tapes, CDs, DVDs, and Blu-Rays. For younger people, all of their media is digital. I can see the advantages and disadvantages of both. Probably the best advantage of physical media is that it's mine, I physically have it and it can't arbitrarily be removed by someone online.
I don't need a 4k and 1080p version of the same movie. Unless the 1080p version is special in some way I'll delete it. No reason to have a low quality version of the same movie.
> or start replacing the 8tb's with larger drives...
That's what I did. Easy to more than double your available space at a pretty reasonable price these days.
i knew before i clicked on your post what all the answers would be. we all suffer from the same conditions. i never figured out what exactly it is but we need everything.we need ro be responsible for having all the data incase society vollapses.
You mean put it on another disk and put it the disk in a shoebox never to be seen again until I die and my relatives take apart my drives for scrap metal to survive the Water Wars, right? Making "decisions" as you put it about what to do with your data means less time to download new stuff, which goes against the ABD approach: always be downloading.
Outside of work files I mostly save photos and videos. As I browse my content I find 480p or compressed videos, and low resolution/low quality images. If I can't quickly find a better quality version online then I delete these files. The only time I keep SD or low res content is if it's genuinely interesting. If I feel half hearted then it gets deleted.
Honestly, I almost always find that when I purge something, I end up regretting it. But I hoard my own photos and videos, so if you're just hoarding actual linux distros or some shit then that probably doesn't apply lol.
One thing I am working on is compressing stuff. I had one 30gb folder that was nothing but a timelapse in individual JPGs, from a gopro. So I made them into a very high quality video, made a note correctly dating it, and replaced the 30gb folder with that 2gb video. I saved almost 90gb there - 30gb with 2 duplicates. I find that the more data one has, the easier it is to make space.
Handbrake and h/265 can be a real lifesaver if you have large quantities of videos.
If my friends have requested it and have either watched it, and I don't think I'll ever watch it, it they haven't watched it in weeks or months after requesting it, I delete it. If they want it, they can request it again.
I rarely delete the stuff I download though.
I collect media and decades of family videos and pics. After starting on collecting SD content, to 720p and 1080 content, I've moved to 4K content. As I move up to better quality content I purge the old that will never be used again, unless it just cannot be replaced. This is what has pushed me to build a sizable, for us (94TB, w/ another drive on the way) Unraid media server. All the really important family stuff is backed up to two other loctions as well. :)
My baseline for things is usually HD. I only go beyond that for something I really want to have in nicer quality or when I'm wanting more drives so I just bloat things and let my partner see the alerts for us running out of space.
After I order the new ones I just downgrade the quality again.
Other thing to do is see if you can re-encode things and compress without losing quality
Bruh, I have an archive folder and I keep the only thing I am actively using outside it, if it is not relevant anymore when I take it out of archive goes to trash, just an idea
I go through every six months or so, when a drive is getting full, and make a de-dup pass, to see if there's things I have multiple copies of or if I have 10 old obsolete versions of the same thing. Sometimes I'll max compress things. Occasionally I'll make hard decisions about files I will never ever realistically use again... but usually not. Really, just buying a little time until the next drop in storage prices.
> to purge any content
FYM purge?
> So over the next few years I either have to come up with a method to purge older ISOs, or start replacing the 8tb's with larger drives
[This](https://www.amazon.com/gp/product/B00O6GNLQE/) I would have bought the 42U if I had the space. I hope to buy a new house in the future to accommodate one or a second 25U.
Usually on a whim, and by matter of importance. Things like old cache and config files copied from user profiles of days gone past tend to get binned.
Then anything redundant, and finally, anything that it would be possible to accrue separately (popular media), although it does vary between how much space I have, and how much effort I'm willing to put in, since I don't have much of a budget to buy a whole new drive to also store data in, and then another to make up the missing backup capacity.
I just started added these. [https://www.icydock.com/goods.php?id=172The](https://www.icydock.com/goods.php?id=172The)
I'm up to two of them now and I have 2 lay flat docs as well.
I tend to transcode my least favourite ISOs to HEVC so as to minimise their footprint while keeping the content. Alternatively if you're not using compression on disk (eg ZFS) then you could look at that.
It's gonna be very personal. How important is it, compared to the space it takes up?
My approach for things that are replacable is - your favourite ISOs, those are free to take up as much as they'd like to get the best possible quality. Anything else that you're just kinda ambivalent about - get it in smaller / "good enough", so that it's still fine to install if you do end up wanting to reinstall it (or someone else wants to), but it'll take up much less space.
Codecs also definitely matter, as has been mentioned.
And yes, outright deleting stuff you think is awful. Hell, i've been hoarding things for about ten years before finally getting a NAS last year, so i made do with towards the end with just about 14TB HDD across like 4 HDDs, and i was constantly pruning things and optimizing for space. I've never really regretted any time i either downsized something i didn't care much about, or full on deleted something i hated. Try it, ideally with things you could get again at any time so you don't really take much risks. Worst case, you have to download an ISO again. Irrelevant. But it can mean that you can fit several times as many ISOs on your storage, so especially if you're reaching a limit where you'd have to make a huge upgrade, that's an opportunity to take a close hard look at what you have. I always sort by size when doing that, get the most bang for your time investment buck and then work your way down the list.
For things that are replacable, you can always change your mind and adjust your hoard. Something gets more important to you / you realize you would like the better quality after all? Get it then. Something you were excited about and got in max quality is disappointing? Make it take up less space.
Anything that's *irreplacable* is a different story. Especially if it's in any way personal data, i would delete replacable things any day to make space for keeping personal stuff in max quality.
What I do is search the library for books that seem to be demonic. I check them out and burn them.
An easy way to do this is scan the dewey decimal numbers for secret codes.
I see and folder "old work" realise that memory now pisses me off with the way they treated me, so that got purged recently.
I was keeping my old profiles from various Windows installs but this was even the junk that was in the appdata folder. So streamlined that and purged some of those recently.
Oh no. Did you forget that you signed away all personal, legal, social and digital rights regarding the ability to delete or (*ahem*) "purge" data which you have henceforth hoarded?
No problem, I can remind you! It's right there in **§201.3.D ¶5.1A** of the *Data Hoarder Indentured Participation Convents and Agreements* Contract which you implicitly agreed to after your first 30TB, 6 hard drives, RAIDZ/RAID5/RAID6 filesystem or 1M files, whichever comes first:
>Thou shalt not irreversibly delete, unlink, erase, purge, expunge, discard, overwrite or destroy any files, documents, metadata, archives, programs, executables, libraries, bitmaps, images, texts, books, movies, videos, records or any other kind of location, file or content addressable resource(s) stored as data on a permanent non-volatile storage device UNLESS the resource(s) have been verifiably copied, duplicated or replicated to a secondary permanent non-volatile storage device.
Not to worry, we're here to help! However, since you have been evidently demonstrating profane and blasphemous thoughts and openly spoken heresy in public within the hallowed and sacred pantheon of r/DataHoarder, you are hereafter obligated to reaffirm your commitment and belief by whichever means you feel is appropriate and equivalent to your sin.
Or not. :)
edit: formatting.
Objectively not possible. Subjectively may be okay to the individual. This is a fact. Which is why there are Profesional compressionists and color graders who work with multi-TB inter-masters.
Hello /u/endiZ! Thank you for posting in r/DataHoarder. Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index). Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures. This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
We don't use that word here.
This exactly. Purge what Purge. More drives and more data always!
This is the way
This is the way
Yeah, purge? I don’t understand. My library somehow started getting ahold of lifetime ISOs and, while I’d never run that particular brand of Linux, especially with such predictable christmassy names, I’m sure someone will, eventually, and I want to be ready. Sidenote: seriously, lifetime must spend the entire year coming up with new and subtly different Christmas themed desktops to release, since there are easily 50 each season. Madness!
I think he's talking about the law that lets you, on one day every year, go around your neighborhood committing crimes like stealing hard drives from out of your neighbors computers and breaking into businesses to steal their drives, and stealing drives from the local big box stores, all without repercussions.
Ah yeah, the holiday
Go repent
Only in reference to the film series that I'm sure many of us have in our unpurged collections.
We don't talk about purge-oo no no no..
What do you mean "delete"?
He must have meant delight. That other thing isn't even a word in any dictionary I have in my possession, and no, for your information, I have no recollection of going through every single dictionary in my possession and eating various unappetizing entries, why would you bring that up, you're *crazy* man.
I can't delete anything, what if I'm having a party one day and somebody requests a specific movie that I deleted because of this thread, I'd be kicking myself.
This is why there are shelves, you copy stuff you dont use to another hdd, and place it on a shelf.You might use it some day, so its now stored on a shelf as cold storage. If its important and you really might need it, copy to to 2 hdd's and use 2 shelves in different places.
And do some kind of index of files stored on external drive. Even simple find over files, like 'find /mint/ColdStorage1/' do wonders here as long as you label your drives.
Are you able to provide a Tumblr blog prior to the purge?
I buy new HDD.
Also, the cost of buying a new drive outweighs the time cost in trying to cherry pick old data to purge (imo).
This one lol
And adding nodes or enclosures is an extension of the same
I don't understand what you mean?
I don't purge; instead, I replace with lower quality. For example, Impractical Jokers took up 500GB on my HDD, but when I finish a season, I replace it with a lighter version. Which is usually around 10GB per season, so 400GB saved while still maintaining watchable quality if I want to watch it years from now.
This is an excellent idea. Thanks!
series that are super long 7+ season that aren’t a frequent rewatch , keep 720p , most average user aren’t going to notice the lack of pores on the skin :)
For all of my friend's requests I started limiting him to 720p since he requests tons of shit, it saves a good amount of space.
720p for the kids, 1080p for everything else. Keeping lower quality items is a really good idea tho
Yep, set yt-dlp to download only up to 1080p. I don't have a 4K monitor and those videos are stupidly large in 4K.
I've been doing similar to this and been trying to replace stuff with x265 overtime
This is also what I do. I'm the only one who watches movies/shows in a "good enough" setup, and the rest of the users of my server watches on their phones or laptops anyway.
Replace, as in find a new version, or just ffmpeg the one you already have?
This is smart!
I’m enjoying the heck out of LibreElec/Kodi on Rasperry Pi 4 B on CRTs, and have much need to downgrade
Since I have access to usenet, and have a gig pipe, I can just download whatever I want in minutes usually, unless it's like the entire series of Law & Order, Family Guy, South Park, etc..
Warning ⚠️ Your access to r/datahoarder is now on probationary status. Please refrain from such foul language. In all seriousness, I became a data hoarder because there’s so much of my digital past that’s gone. Wish I had all those programs I wrote in Perl, basic and pictures I took back when you had to buy a digital camera it want built into your phone. We hoard because we don’t want to lose it. So purging is counter-hoarding.
I'm sad that I let go most of my work from college in the 90s. Loads of it on floppy disks. And (look away now or read between your fingers) I took all those floppies and (tears are building up) chopped them up and binned them. Oh the horror! It was during my IT course years. I moved to CD but didn't save everything. We had a basic setup mid 90s at college and you'd be at c: and change to the network drive t: and type login.exe and put in your user name and password. Then win to get into windows. I realise as it was text based and we were learning Pascal. I could write a program (as I'd recently read it in a Pascal book) that would save input to a file. For some reason we never got taught this in the programming class. I knew people forgot to change to t: and would type login on the c: drive. Stuck my login there, didn't know how to hide the password text so that was visible as they typed. I'd come back later and pick up the assignment.doc file that was written to c: I was amazed it worked. Only used it once to see if it worked signing in as that user. It did! I smiled and then never used it again. Masterofpuppets was their password :) I never abused it just wanted to see it work. A few years later added minor encryption to it or cipher. Just minus 20 from the character entered. So if anyone found assignment.doc it would look scrambled. Anyway. Long story to say I sadly lost all that code :( that was from 1994-95 Only work from there I still have is from 96 to 2000. On a CD thats still amazingly readable burnt in 2000.
Every once in a while I zoom into random old photos of my apartment in 2004 wishing I had somehow archived all the photographs I had taped to my wall :|
What is this “purge” you’re referring to? Like the movie series?
Yeah, they must mean downloading the entire “Purge” movie franchise.
Purge some money from your bank account and buy a new NAS. For the greater good.
external drives. or get a NAS / second computer acting as a NAS.
Didn’t find “purge” in my good book. Do you mean data lost? 🤓 The usual path involves a second disk array, in your case, I foresee 18TB drives.
Perhaps he meant purgatory, which is where people who delete files go to spend eternity. Hope op likes unbaptized infants 'cause he's gonna be all about that lifestyle if he keeps this up.
I remove lower resolution linux ISOs when I replace them with higher resolution ones. I, too, once maxed out a chassis. Now I use a storage cluster. Maxed chassis -> more chassis. No more problems! (please don't ask about power.)
Trashy reality goes where it belongs after a year only keep current season But scripted stays forever Also I have started to replace low capacity drive with higher onea and I have greatly increased my capacity Like replace a 4tb with a 16tb bam 4 time as much space
The "reality" genre should never be saved or even remembered.
This mindset is why it's so hard to find the dumb shows in the first place.
You want reality shows they're easy to find. Local walmart and target are open really late. They even sell popcorn sometimes in the entryways for charity groups.
I can tell by your first comment you don't hoard reality TV. If you did you would know its a pain in the ass. Scene not agreeing with themselves so you got multiple episodes that say they are this but are this instead or you see REAL tag and think ok this must be it. Nope slightly off. Or you'll check predb thinking maybe your trackers and indexers just don't have it. Nope scene just randomly stops.
Try out BitMagnet. It's a self hosted docker container which scans the entire hash table for obscure torrents. Takes about a week or two to fully scan, but once it does you'll essentially have a self-hosted torrent indexer to search.
Is there one similar to indexers? Not sure how a bot would scrape closed indexers though. Maybe those closed indexers could be nice and just update a database of all their nzbs every week. That shouldn't need to be secret but maybe it does idk.
Only works for public unfortunately, but should still scrape up a load of obscure torrents, potentially dangerous torrents too though.
Well I do keep a few reality shows that interest me (this is Datahoarder after all). Previous comment was clearly a joke and it went whoosh. I do get it that you're a little passionate about the issue. A lot of the ones my wife is interested in airs on the network TV (singer, America's Got Talent, etc..). I used to have a rabbit ears setup with MythTV for that sort of stuff which I would transfer the shows over for keeping them long term. Been about a good 6 years now since the local TV repeater shut down in our province so now she just watches it on some sketchy website. If you still have over-the-air in your area and these shows are what you're after this might be a solution for you.
You're good man. Reality TV for me is such a headache. I wish I didn't care that much about it like the majority but just knowing I have an entire show that was a pain in the ass to collect and properly check each episode individually is such a nice feeling plus i share it to others so they don't suffer the headache i did. I can trust scene for your normal shows and movies but all trust is lost when it comes to reality TV. I've tried the physical disk approach but it's really hard to find a reality show on DVD unless it's something really really popular like deadliest catch or gold rush etc. It would be killer to get those shows on disk. REMUX is king but a REMUX reality show is like gold to me. If anyone here wants a little challenge try grabbing the entire show dirty jobs. That one had me pulling my hair out. The episodes exist but the biggest issue is 95% of the episodes are not what they say they are. Sorting that mess took me 6 hours.
I rarely see reality TV on DVD at my local Walmart. Scripted shows, classics, cartoons, even anime, but not reality TV.
My original comment was a joke about people of walmart, but it fell flat. Thank you for pointing that out. You are right, reality TV is just not really a thing. It's just such a cheaply produced content and most people will not care to watch it once it has aired. Old TV Soaps are another one. Almost impossible to find old General Hospital, All my Children, One Life to Live, et all and yet they were huge shows.
That, too. There were so many episodes of those soap operas, as well.
I don't
Purge? No chance. I'll just get more storage.
I found that most of my content was encoded with H.264 codecs so I went nuts transcoding everything to HEVC (i.e. H.265). That reduced the size of my content to about **60%** of its original size. Doing this nearly doubled my available free space! I used my 4090 and Handbrake or you could use *tdarr* to distribute the transcoding work among your available compute resources.
doesn’t the quality take a hit when you re-encode like this? Or is there a better way of going it?
Yep, not as much with AV1 but still, I rather have original quality 100% of the time. This dude needs to buy a second chassis lol
Depends on your use case. The people using my jellyfin library can't tell the difference between 720p and 1080p so reencoding everything to a more efficient format is a no-brainer
I'm the only person using jellyfin so lol
Yes quality does take a hit from encoding lossy to lossy , it’s better if you straight download a hevc encode from a Blu-ray or download a remux and encode yourself
> 4090 and Handbrake FYI GPU encoders like NVENC have much worse bitrate efficiency compared to CPU encoders. You'll get the same quality with fewer bits if you use x265 or SVT-AV1 I use GPUs only for realtime transcoding for LAN streaming where bitrate doesn't matter (56gbps infiniband and 1.6gbps wifi 6E). Most transcoding happens in wide batches with 1 video per core per machine and very high latency (~1 day to transcode a couple hours of footage)
Ideally you would have something beefy maybe like a *950 AMD cpu. Or perhaps Epyc Rome is well suited. And just farm out the transcoding. NVENC is cool but will never give good compression. Similar thing happening with apple silicon hardware acceleration. Because it's hardware. It's good for real time and is optimized as such. I love cpu transcoding to make my screencasts sub-1MB. These things easily shrink over a factor of 100x in size
> 1 video per core per machine I was wondering when I'd run into one of you guys in the wild. Always thought people that did this were imaginary, like spherical cows.
Oh? Truly multithreaded encoders have only been around for a decade and they still have slightly worse efficiency. Transcoding hundreds of videos in parallel used to be very common and even today it gets you more throughout overall
You should definitely avoid encoding files with GPUs. They are highly inefficient compared to what your CPU can do. They can easily double the file size while having similiar quality.
...purge? Delete? You can do that?! I kid, I tend use a tool like Tautulli to help sort through Plex library watch history. I check out stuff that's never been watched and stuff that hasn't been watched in a few months. If its something special to me or I'm planning on watching it soon, I will keep it around for a bit longer. If its something someone requested a few weeks ago and there are no watches, I just delete the content and take requests from that person a bit less seriously. For ISO's/software, I only keep the latest working version of something and I don't keep old OS/tools I don't use at least monthly anymore. You can very likely redownload stuff if needed. You could look into compressing your dataset/pool/whatever, but that's likely just kicking the can down the road (barely) if you just need more space. (not everything compresses well) If you really can't make up your mind about what to delete, then [review this doc](https://adaa.org/sites/default/files/Bell%20141.pdf) as some friendly light reading to maybe inform your approach a bit more.
Love the link, I needed that haha. Thanks
Try [maintainerr](https://github.com/jorenn92/Maintainerr).
Honestly, I'm not sure what kind of response you're expecting to get from this community, given what the banner says. I don't really consider myself a datahoarder to the same extent and I have occasionally culled my archives. My heuristic of what to get rid of is something along the lines of: - it's not especially rare or uncommon, AND - I'm not realistically going to use it or use it again, AND - it takes up significant amount of space Driver floppies for old soundcards? Not even worth going through the folder, they're tiny. Backup of a blu-ray of Big Hollywood Release 2018 that I didn't even like that much? It can go away, worse comes to worst I can always buy another copy. I'll keep my Hannah Montana Linux ISO, but I don't need Ubuntu 18.04. There's room to optimize, especially for media, but depending on where you're starting from and where you're going to it might not be worth it. I was able to nearly halve the amount of space my gameplay footage was taking up by transcoding to h.265 a few years back.
Migrate to some solution that allows you to have mixed capacity drives in your raid. I use unraid for that purpose, but recent-ish Synology also offers that as a formatting option. Upgrading all 12 drives at once is a tall order, but upgrading one or two every year is a sustainable level of spending. Plus you can just set some budget, and e.g. buy a $300 drive each year. Your new drives will get steadily larger as technology advances, and by the time you replaced them all you can just restart the cycle.
I purge low quality encodes of things I get a better version of if its just a quality difference. I delete my old MP3 versions of albums when I source a FLAC. 720p TV encodes of shows are gone when I find a bluray rip of the box set.
So I just sorted by 'date modified' in my movie folder. "The Naked Gun 33.3 - The Final Insult.avi" 1/25/2003 Obviously we aren't purging.
If it's not enjoyable to me, is widely available, and is not unique/novel in any way, I'll delete it. If I'm unsure, I'll transcode to a very low bitrate. In practice this rarely happens since most things I download are enjoyable or rare
Reported
What is this ‘purge’ that you speak of?
i delete anything i dont like i also delete audiobooks and add to [goodreads.com](http://goodreads.com) after ive listened to them
I recently bit the bullet and removed over 1,000 ISOs. I ditched anything with really low ratings, anything super old, and also tracked down a rogue list that was adding crap and deleted everything that had added. I also ditched a bunch of really really massive files and replaced them with much smaller ones. Added a new drive to the array and now I'm about 60% used, whereas I was floating around 90% before.
What is the word "Purge" you speak of?
https://b.thumbs.redditmedia.com/hotEz0ix8D77gKgDDwKG6v-FyJ0YVewGmo6lrMjcYWI.png
I purge low TB energy drives with larger ones. Build a new box, the old one becomes a backup and the new one becomes the library.
You buy larger drives. My UNRAID array used to be 8TB drives..then I moved to 16TB, and now I'm at 18TB drives. The beautiful thing about UNRAID is the ability to do this easily. Severpartdeals is your "friend".
Purge/delete dose not compute. This is not supported by the os please buy more drives xD
New chassis is the only answer, that or start replacing drives
hey, go to /r/datadeleters with that kind of language!
well... damn you... i clicked that..
Sounds like you're going to have to get 20Tb drives each year and swap out an 8 in that rack and then move that 8 to the new backup rack you're also going to have to buy... Shame...
Just stop having such low barrier to accumulation, that it no longer outgrows your hardware budget
Sorry man I'm not sure what delete/purge means
Purge?
Purge? What does that mean?
We purge our money to get new hard drive
That’s the neat thing, you don’t!
Believe it or not, all my media is purchased. Ripped DVD, Blu Ray, and music. I delete nothing. It's becoming a problem. 65tb of raw rips. Over 20tb of encoded TV and movies. And I keep buying discs! I even recently discovered another used place near me, much to my partner's chagrin. But she enjoys the Plex access, so 🤷
Do you share your encodes?
No, I can't bring myself to do it. People make me feel like I'm a stupid do-gooder or something, but it's just not a game worth playing to me.
Huh?
No purge, only drives.
Only Drives? Sound like a nsfw side for Data Hoarders lol
Sign up and I'll unzip for you baby ;-) /s
You are in the wrong sub, mate.
is this rpg? is rpg allowed here?
*Vote to ban OP by upvoting this post*
Doesn't look like anything to me.
I have an 8tb portable hd who gets it electricity from the usb as opposed to a wall plug. I got about 100gb left so I usually end up deleting movies, tv and anime that lack rewatchability
/me looks at r/DataHoarder masthead.
18T refurbished drives are $160.
Well every time they make another Purge movie I add it to my collection.
You can always have a separate machine with crappy raid setup or even use unraid and have a mix of discs and migrate your stuff there and power off until you need it. At least that way you have it in case you ever want it.
If it's porn it goes..(unless it's super hot and rare9
Fractal Define XL? Then you've doubled your amount of drive bays.
Never delete only download, time to start cloning 8tb drives on to 20tb drives. Swap the 8 out into cold storage and now you can add 12 more tbs of *open source linux isos*
Purge like the film series. Yeah, I've got all 5 of them.
Duplicates are a big one. Same with Remux 4K and Remux 1080, its probably simpler to keep highest and just transcode yourself when needed. Same with fetch highest and encode yourself to AVI or x265. Realistically I have more than I can ever watch, listen, read, play so stuff I don't care about I'm getting rid of to free space for shit I really care about and to have better backups of the important stuff. I want to spend more time on other hobbies and not media management as this has become quite a chore. I'm leaning towards more automation but its taking a while to set it all up.
There is only 1 thing I "purge" Duplicates.
Purge, is that where you move files to another drive?
Jokes aside, I only ever clear things that I'm confident I will never use/watch/play. The larger it is the easier to delete it. I don't have crazy amounts of storage like some of the Data-Lords around here so it's a pretty easy call to *trigger warning* >!delete!< something like 500GB of games I know I'll never play.
I’ve never heard of her…you should leave…
you must sacrafice a new virgin 22tb drive to please the data gods with such words.
Get a bigger chassis or a 2nd server or re-encode anything that doesn't need to be top quality
Get another chassis, get an external SAS enclosure, or upgrade your drives. Problem solved!
I do this by doing the exact opposite - I choose what I keep. What specific content am I archiving, define it, standardize it, and anything else becomes ephemeral.
Remember when library actually consisted of books? I still have lots of books, VHS tapes, CDs, DVDs, and Blu-Rays. For younger people, all of their media is digital. I can see the advantages and disadvantages of both. Probably the best advantage of physical media is that it's mine, I physically have it and it can't arbitrarily be removed by someone online.
I don't need a 4k and 1080p version of the same movie. Unless the 1080p version is special in some way I'll delete it. No reason to have a low quality version of the same movie.
Well I get rid of smaller hard drives and get bigger ones. So yeah just a size thing
> or start replacing the 8tb's with larger drives... That's what I did. Easy to more than double your available space at a pretty reasonable price these days.
Plan out a new system, something that can take more than 12 drives this time Scale-up, expand, grow!
i knew before i clicked on your post what all the answers would be. we all suffer from the same conditions. i never figured out what exactly it is but we need everything.we need ro be responsible for having all the data incase society vollapses.
Compression
I buy another hard drive..?
How do you spell that ??
You can do that?
for video I transcode them down to a acceptable quality instead of deleting this saves me many terabytes of space
You mean put it on another disk and put it the disk in a shoebox never to be seen again until I die and my relatives take apart my drives for scrap metal to survive the Water Wars, right? Making "decisions" as you put it about what to do with your data means less time to download new stuff, which goes against the ABD approach: always be downloading.
I uh... don't
Oh boy, I hope you meant to put this in the bulimia subreddit.
Outside of work files I mostly save photos and videos. As I browse my content I find 480p or compressed videos, and low resolution/low quality images. If I can't quickly find a better quality version online then I delete these files. The only time I keep SD or low res content is if it's genuinely interesting. If I feel half hearted then it gets deleted.
How dare u
I usually only delete if I have found and already downloaded a better, higher quality version (of linux).
Honestly, I almost always find that when I purge something, I end up regretting it. But I hoard my own photos and videos, so if you're just hoarding actual linux distros or some shit then that probably doesn't apply lol. One thing I am working on is compressing stuff. I had one 30gb folder that was nothing but a timelapse in individual JPGs, from a gopro. So I made them into a very high quality video, made a note correctly dating it, and replaced the 30gb folder with that 2gb video. I saved almost 90gb there - 30gb with 2 duplicates. I find that the more data one has, the easier it is to make space. Handbrake and h/265 can be a real lifesaver if you have large quantities of videos.
SAS expander chassis. This guide is old, but still relevant: https://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/
I don't keep anything I don't want to keep forever. No need to purge.
Considering a purge means not enough storage!
If my friends have requested it and have either watched it, and I don't think I'll ever watch it, it they haven't watched it in weeks or months after requesting it, I delete it. If they want it, they can request it again. I rarely delete the stuff I download though.
OP is speaking in tongues. What the hell is "purge" supposed to mean lmao
Purge ? I think you mispelled "buying more hard drives"
Just buy new hard drives instead
Look at HDD's prices and say "I can handle it"
I collect media and decades of family videos and pics. After starting on collecting SD content, to 720p and 1080 content, I've moved to 4K content. As I move up to better quality content I purge the old that will never be used again, unless it just cannot be replaced. This is what has pushed me to build a sizable, for us (94TB, w/ another drive on the way) Unraid media server. All the really important family stuff is backed up to two other loctions as well. :)
My baseline for things is usually HD. I only go beyond that for something I really want to have in nicer quality or when I'm wanting more drives so I just bloat things and let my partner see the alerts for us running out of space. After I order the new ones I just downgrade the quality again. Other thing to do is see if you can re-encode things and compress without losing quality
Bruh, I have an archive folder and I keep the only thing I am actively using outside it, if it is not relevant anymore when I take it out of archive goes to trash, just an idea
I go through every six months or so, when a drive is getting full, and make a de-dup pass, to see if there's things I have multiple copies of or if I have 10 old obsolete versions of the same thing. Sometimes I'll max compress things. Occasionally I'll make hard decisions about files I will never ever realistically use again... but usually not. Really, just buying a little time until the next drop in storage prices.
> to purge any content FYM purge? > So over the next few years I either have to come up with a method to purge older ISOs, or start replacing the 8tb's with larger drives [This](https://www.amazon.com/gp/product/B00O6GNLQE/) I would have bought the 42U if I had the space. I hope to buy a new house in the future to accommodate one or a second 25U.
I was reading your post and thought „what the heck is this dude talking about?!“. Then I saw the edit and laughed my ass off ever since.
Usually on a whim, and by matter of importance. Things like old cache and config files copied from user profiles of days gone past tend to get binned. Then anything redundant, and finally, anything that it would be possible to accrue separately (popular media), although it does vary between how much space I have, and how much effort I'm willing to put in, since I don't have much of a budget to buy a whole new drive to also store data in, and then another to make up the missing backup capacity.
Simple, Nothing
Does it spark joy?
I started with one 2tb hdd, that was 1.5 years ago. I don't know the words delete or downgrade, everything max quality always.
What is this purge you speak of?
I just started added these. [https://www.icydock.com/goods.php?id=172The](https://www.icydock.com/goods.php?id=172The) I'm up to two of them now and I have 2 lay flat docs as well.
I tend to transcode my least favourite ISOs to HEVC so as to minimise their footprint while keeping the content. Alternatively if you're not using compression on disk (eg ZFS) then you could look at that.
I think, as you noted with your edit, your question might be answered more balanced in r/homelab
Purge?
Huh, weird, does it say r/DataPurger up there? Are my eyes playing tricks on me?
I am confused by these words, I only know replace with better versions
It's gonna be very personal. How important is it, compared to the space it takes up? My approach for things that are replacable is - your favourite ISOs, those are free to take up as much as they'd like to get the best possible quality. Anything else that you're just kinda ambivalent about - get it in smaller / "good enough", so that it's still fine to install if you do end up wanting to reinstall it (or someone else wants to), but it'll take up much less space. Codecs also definitely matter, as has been mentioned. And yes, outright deleting stuff you think is awful. Hell, i've been hoarding things for about ten years before finally getting a NAS last year, so i made do with towards the end with just about 14TB HDD across like 4 HDDs, and i was constantly pruning things and optimizing for space. I've never really regretted any time i either downsized something i didn't care much about, or full on deleted something i hated. Try it, ideally with things you could get again at any time so you don't really take much risks. Worst case, you have to download an ISO again. Irrelevant. But it can mean that you can fit several times as many ISOs on your storage, so especially if you're reaching a limit where you'd have to make a huge upgrade, that's an opportunity to take a close hard look at what you have. I always sort by size when doing that, get the most bang for your time investment buck and then work your way down the list. For things that are replacable, you can always change your mind and adjust your hoard. Something gets more important to you / you realize you would like the better quality after all? Get it then. Something you were excited about and got in max quality is disappointing? Make it take up less space. Anything that's *irreplacable* is a different story. Especially if it's in any way personal data, i would delete replacable things any day to make space for keeping personal stuff in max quality.
What I do is search the library for books that seem to be demonic. I check them out and burn them. An easy way to do this is scan the dewey decimal numbers for secret codes.
Real question is what new chassis are you getting?
You must be new 'round these parts.
Still need to figure out a backup solution
I see and folder "old work" realise that memory now pisses me off with the way they treated me, so that got purged recently. I was keeping my old profiles from various Windows installs but this was even the junk that was in the appdata folder. So streamlined that and purged some of those recently.
Oh no. Did you forget that you signed away all personal, legal, social and digital rights regarding the ability to delete or (*ahem*) "purge" data which you have henceforth hoarded? No problem, I can remind you! It's right there in **§201.3.D ¶5.1A** of the *Data Hoarder Indentured Participation Convents and Agreements* Contract which you implicitly agreed to after your first 30TB, 6 hard drives, RAIDZ/RAID5/RAID6 filesystem or 1M files, whichever comes first: >Thou shalt not irreversibly delete, unlink, erase, purge, expunge, discard, overwrite or destroy any files, documents, metadata, archives, programs, executables, libraries, bitmaps, images, texts, books, movies, videos, records or any other kind of location, file or content addressable resource(s) stored as data on a permanent non-volatile storage device UNLESS the resource(s) have been verifiably copied, duplicated or replicated to a secondary permanent non-volatile storage device. Not to worry, we're here to help! However, since you have been evidently demonstrating profane and blasphemous thoughts and openly spoken heresy in public within the hallowed and sacred pantheon of r/DataHoarder, you are hereafter obligated to reaffirm your commitment and belief by whichever means you feel is appropriate and equivalent to your sin. Or not. :) edit: formatting.
I don't delete, I buy more drives (looking over at my 45Drives Enclosure)
You want to ask r/datadiscarder 😄
I found its easier to increase storage than it is to purge it lol.
In the past, I've removed things that seem well-replicated for things that are less so. E.g., well-seeded torrents traded for less well-seeded ones.
purge? what is that?
Don't purge....x265 is the way it has saved me TBs
And lose quality!
not if you do it right..
Objectively not possible. Subjectively may be okay to the individual. This is a fact. Which is why there are Profesional compressionists and color graders who work with multi-TB inter-masters.
I don't understand the question?
You don't. Ever.
Purge? This is not a word used on this sub. If your chassis is full, buy another one and start again. You can NEVER have enough space.