Marshall Brain's Blog

---

This page by Jim Gray offers this interesting quote:

Computing is free. The world's most powerful computer is free (SETI@Home is a 54 teraflops machine).

I have been a long-time fan of SETI@home, and SETI@home is definitely a computing problem that needs every spare CPU cycle it can get. When you run the SETI@home screen saver, your computer downloads 400K or so of data and then spends the day grinding away on it.

Another effort that is trying to capitalize on the "donate your computer's idle cycles" phenomenon is Grub. The Grub screen saver is the "spidering" or "crawling" half of a search engine. When you run the Grub screen saver, your machine downloads a package of 500 URLs. Your machine then goes out and reads those 500 pages, and sends back the keywords it finds to the Grub servers. The Grub servers then use the data that everyone has contributed to implement a Google-like search engine.

The donation that people are making to Grub is different from the one they make to SETI@home -- with Grub, you are not donating much CPU time. Instead, you are donating your bandwidth. With a fast broadband connection, my machine can easily spider half a million URLs per day as a Grub donor. It is estimated that there are something like 10 billion pages on the Web. If 20,000 people donated half a million spidered URLs per day, then it would be possible to crawl the entire Web every single day. If you assume that the typical Web page has 10,000 bytes of text on it, that implies that Grub needs 100 Terabytes of bandwidth donated every day to spider the entire Web daily.

Grub offers people an incentive for donating their bandwidth. Donors can direct their machines to spider their own Web sites. This allows donors to make sure that their personal/corporate Web sites get spidered every day. The results are then available at the WiseNut search engine. If millions of people were using WiseNut in the same way that millions use Google, that would be a valuable incentive. Time will tell if WiseNut develops a significant following. This article from Wired News points out that bandwidth and spidering are only part of the equation for a successful search engine. The other half is the algorthms in the search engine that deliver the results. So far, Google is the best at creating good result algortihms.

When you think about it, Napster, Kazaa, Gnutella, and other file sharing programs are donation systems as well. With file sharing, people donate their CPU cycles, bandwidth and hard disk space so that others can download the files they need. The most brilliant part of Napster is the fact that one person -- Sean Fanning -- could build a system able to support 60 million visitors a month downloading terrabytes of data every day. He spent very close to zero dollars on hardware and bandwidth because all of it was donated by the users. It will be interesting to see what other uses people come up with for donated CPU cycles and bandwidth...