So… I was having a discussion with a copyright classmate about the legitimate uses of bittorrent, explaining that (like any good protocol) torrent can be used to transfer both legal and illegal bits.
screenshot by dantekgeek, used under CC-SA
It struck me that many would probably be able to spare disk space for a pile of iso and some upload. (I certainly can.) But there is no way, that I can see, to automatically ‘mirror’ the torrents of the major distros- all the torrent pages seem to assume that you just grab a single iso that you need, and then leave your client running; no information on automatically grabbing new torrents; shutting down old ones; etc. It seems like there are many people who could be talked into more pro-active mirroring if there were a scriptable way to do that, like the elaborate mirror scripts that already exist for ftp/http.
Do such scripts exist for torrent? I’d like to use some of them if they do :)
[Long term, without having given it much thought, this also seems like a good idea for rpms/debs; yes, there is some overhead for smaller files, but using a protocol that automatically load-balances and mirrors seems like it would be a good idea. It took me a full year on columbia campus before I noticed that I had a local mirror; I like the idea of using a protocol that would figure that out for me, and automatically allow me to help out/participate as well. But like I said, I haven’t given this much thought.]
Jeremy Katz: Firefox 3 in Fedora – Dec 20, 2007 Chitlesh GOORAH: Fedora Electronic Lab for 100$ ??? – Dec 20, 2007 Behdad Esfahbod: GNOME Accessibility to Get a Boost – Dec 20, 2007Luis Villa: automatically ‘mirroring’ distro torrents? – Dec 19, 2007 Bastien Nocera: GUADEC vids – Dec 19, 2007
It would be cool if there was a slick little tool where I can just go “Donate X gb of space to Freedom Torrent” or whatever, and it would just automatically manage everything for me, so I don’t have to prune old releases and manually add new ones.
Someone could just manage the “upstream list” of current torrents and users wouldn’t really have to care other than managing the amount of space/bandwidth they want to donate.
There is something called debtorrent. I haven’t tried it myself, but I think it works pretty well.
http://debtorrent.alioth.debian.org/
the overhead for small files in torrent is pretty bad.
Fedora’s Mirror Manager works with the default mirrorlist url generator to help make use of local mirrors when available. Local admins for a netblock can define a local mirror location for the fedora repository and every client from that netblock will then be handed the address of the local mirror from the Fedora mirrorlist generator.. with no local configuration necessary.
As long as the local miror admin keeps their mirror in sync, it will be the preferred mirror for the local netblock for which they are providing the local mirror.
Check out mirror manager.
-jef
Many Bittorrent clients support discovering torrents via RSS, often through a plugin of some sort. So if the distro / projects you wanted to mirror published their list of torrents in an RSS feed it would be fairly easy to pick up new releases.
http://torrent.fedoraproject.org publishes an rss feed of all of its torrents. This was done to handle auto-mirroring of our torrent. A number of torrent clients obey the rss format. A contributor to fedora told me about the format and I wrote the script that adds them to it automatically.
A lot of torrent servers have this already, I think.
You’re asking for a cloud file system. It is a well know concept but it has never been implemented on a large scale. Cloud file systems always get filled with illegal content and that brings on the liability question. Mirrors are a controlled way of doing a primitive cloud file system.
For an interesting intellectual exercise imagine a world without copyright on digital representations. In this world the cloud file system is the perfect solution. Microsoft loads a new Windows onto the cloud disk drive and everyone magically starts booting from it. All the books, music, and images of the world are on anther cloud disk. Everyone’s disk becomes a cache and the network sorts out the optimal place to redundantly store everything. All information is stored in standardized form and perfectly indexed.
But alas digital representations have copyright and we have this world of billions of private disk drives. Illegal copies are retransmitted trillions of times over the internet because of the liability of storing them. Everything is a confusing mess since all of the right owners don’t cooperate with indexing – that’s called marketing – make your product more visible and hide the others.
Remove copyright and allow efficient caching. 90% of the world’s internet traffic would disappear and we would all be able to consume 10x the content with out generating network load.
Many clients offer to watch a certain directory for torrent files and will then watch that directory for additions/deletions of torrents.
So only the ‘how to get hold of the torrents’ part left. There you have all options: rsync, rss feeds, crawler like stuff, etc.
There is not only one way to do it…
No, I’m really not asking for a cloud file system :)
I’ll subscribe to the torrent RSS; it should be better documented on the torrent.fedoraproject page.
Surely torrents negate the need for historical “mirrors” as once a client has acquired 100% of the data, it can happily seed forever.
I don’t see the need for official mirrors, just fast seeds (that Ubuntu somehow seem to manage)
Presumably Ubuntu manages in part by having piles of money. ;) But yes, I agree that (at least in theory) having torrents could make the whole mirror structure mostly redundant, except for those institutions that for some reason need to block torrents.
Torrents don’t remove the need for mirrors b/c torrents don’t work out well for single, small files. Also distributed mirrors deal well for places where a link internally is MUCH MUCH MUCH cheaper than an overseas link.
You could, potentially, do distribution by a network of seeds and trackers in multiple locales but that let’s people get ALL The bits, but not precisely “this bit” of the files hosted on the torrent.