Data hoarding

Table of contents

The old blog that was here under the same domain had also a post on data hoarding. Too bad it was really just a pretentious waste of space and not worth saving. Instead of elaborating on why data hoarders cannot cope with their own mortality or some shit like that, I try to provide some helpful information instead.

Data hoarding tips

  • Make regular backups. If you can, try to make a cron job to run a shell script that runs rsync every week or so and saves the data to another storage medium.
  • Use universal, free, and open formats/codecs. This makes it easier to make it use your files on different operating systems and prevents you from DRM holding your own data hostage.
  • Use a filesystem that works on all the different OSes you need, or what's even better, use NAS where filesystems are irrelevant
  • Get everything where it's feasible as physical media. Books in particular, because those are easier to read with a physical copy and don't depend on electricity. Try to be as low-tech as possible for the data in question. Be careful to not buy DVDs or Blurays with DRM, in that case digital media is preferable.
  • Similar to how you should stay as low tech as possible, try to use as simple formats as possible. Plain text instead of PDF, for instance. If it's not available as such, convert it with pandoc or ffmpeg. This makes the files accessible on simpler devices too.
  • In many cases sorting your files can be automated by querying the metadata with a script (Python and Perl have a bunch of libraries for various file formats like mp3 or flac to use)

On Ogg Vorbis and Plain Text

  • For music and audio books ogg vorbis is by far the best format. It's one of the most effecient (size/content) audio formats, if done correctly it is indistingushible from a good mp3 (lossless is just wasting space for no good reason), it is free and open and widely available on every music player you might want. I highly recommend you use it. Bandcamp offers ogg vorbis too, unfortunately not as much Slsk, but they have a lot of flacs, which you can convert without quality loss.
  • Similarly, plain text is most of the time more than enough. For ordinary books you don't need to have a pdf or ebup. Plain text can be read on any device ever, while having the best (size/content) ratio. Plain text is beautiful and simple, pdf is ugly and complex. You can convert most of your pdfs to plain text with pandoc or a similar tool and edit it afterwards if needed.

Where to get data to hoard

  • Soulseek is good for music (and sometimes also video game ROMs!)
  • Consider USENET (see my guide on here)
  • Private trackers (see https://igwiki.lyci.de/wiki/Private_trackers)
  • Books: Annas-archive, Library Genesis+ (the old Libgen is basically dead, Libgen+ replaces it), Z-Library, pdfdrive, Project Gutenberg, Liber3, OpenLibrary, MemoryOfTheWorldLibrary
  • IRC XDCC (#nibl on Rizon for anime), Doujinstyle for otaku music
  • https://kiwix.org/en/ for Wikis, albeit a lot of Mediawiki instances have a way of downloading the whole wiki already. Some Mediawiki-based Wikis also have a "Download as PDF" feature.
  • Httrack and wget can be used to download entire websites for reading them offline
  • yt-dlp works on more than just Youtube and can download from hundreds of other video sites too
  • gallery-dl is yt-dlp for images
  • Imgbrd-grabber for mass-downloading images from various boorus and Pixiv
ยท 3 min read
../