Click here to go back

Data hoarding

The old blog that was here under the same domain had also a post on data hoarding. Too bad it was really just a pretentious waste of space and not worth saving. Instead of elaborating on why data hoarders cannot cope with their own mortality or some shit like that, I try to provide some helpful information instead.

Data hoarding tips

• Make regular backups. If you can, try to make a cron job to run a shell script that runs rsync every week or so and saves the data to another storage medium.
• Use universal, free, and open formats/codecs. This makes it easier to make it use your files on different operating systems and prevents you from DRM holding your own data hostage.
• Use a filesystem that works on all the different OSes you need, or what's even better, use NAS where filesystems are irrelevant
• Get everything where it's feasible as physical media. Books in particular, because those are easier to read with a physical copy and don't depend on electricity. Try to be as low-tech as possible for the data in question. Be careful to not buy DVDs or Blurays with DRM, in that case digital media is preferable.
• Similar to how you should stay as low tech as possible, try to use as simple formats as possible. Plain text instead of PDF, for instance. If it's not available as such, convert it with pandoc or ffmpeg. This makes the files accessible on simpler devices too.
• yt-dlp, gallery-dl, wget (it can archive entire websites), httrack (inferior to wget due to being more bloated without being more useful), aria2 (it can download torrents), Nicotine+ and slskd (for Soulseek), rtorrent
• In many cases sorting your files can be automated by querying the metadata with a script (Python and Perl have a bunch of libraries for various file formats like mp3 or flac to use)

Where to get data to hoard

• Soulseek is good for music
• Consider Usenet binary newsgroups
• Private trackers (see https://igwiki.lyci.de/wiki/Private_trackers)
• Books: Annas-archive, Library Genesis+ (the old Libgen is basically dead, Libgen+ replaces it), Z-Library, pdfdrive, Project Gutenberg, Liber3, OpenLibrary, MemoryOfTheWorldLibrary
• IRC XDCC (#nibl on Rizon for anime), Doujinstyle for otaku music
• https://kiwix.org/en/ for Wikis, albeit a lot of Mediawiki instances have a way of downloading the whole wiki already. Some Mediawiki-based Wikis also have a "Download as PDF" feature.