This is a question that comes to mind every time I spend a few days focusing on the fediverse. Normally I’m on the microblogging side, but now I have a Lemmy account it might start a proper discussion.

So, to the point, pretty much every fedi platform has similar problems with small servers taking a beating whenever a post goes viral. This ends up costing the server owner a bunch of money trying to keep their server alive while thousands of instances attempt to pull large static files from the original host’s post. This recently instigated this call to action on this forum.

I’ve never seen the question of torrents answered and it feels like a lot of effort and a bit self entitled to get the ear of fedi software devs to implement torrents as a solution, so I’m putting this here.

If media files were made into torrents when a post was being created, an extra object could be added to post objects like

'torrentcdn': {
  'https://imagePathAsKey.jpg': {
    'infohash': 'ba618eab...',
    'torrentLocation': 'https://directlinkto.torrent',
    'webseed': 'https://imagePathAsKey.jpg',
    ...
  }
}

This would not break compatibility as it would just be ignored by anything not looking for a ‘torrentcdn’ object, yet up to date instances could use this instead of directly pulling the static files.

This would benefit instances as when a post goes viral, the load would be distributed amongst all instances attempting to download the file.

This could also benefit clients and instances as larger files like short videos could be distributed using webtorrent, massively reducing the load on server when many people are watching the same video.

Thoughts?

  • steventrouble@programming.dev
    5·
    5 months ago

    Great question! A distributed systems expert is the person you want to ask.

    This sounds like distributed systems 101, specifically the multicast problem, which has a lot of possible solutions, none of which Lemmy is using. In particular, gossip seems like a good candidate.

    Basically, when trying to download a media file from an instance, lemmy servers could ping a few other nodes with less traffic first to see if they have that file before trying to fetch it from the origin server.

    (Lemmy isn’t big enough that we should need to use gossip yet, though. I agree with the other post saying that some basic caching would go a long way.)

    Internally, torrent uses similar strategies to spread out traffic, but because it’s focused on longer term storage of larger files, it would be too wasteful and slow to use for individual png files.