PAULINA STREET JOURNAL
Curing cancer by committee
July 29, 2010

My eldest decided the other day it was time I knew about something about Web 2.0 — Web 2.0 being broadly understood to mean whatever happens next in the world of the Internet. So he presented me with a couple of books plus an essay, which he felt would get me up to speed. Thinking to indulge the lad, I've now read these learned works, and find myself disturbed. For example, is your idea of diversion an evening plopped in front of the tube? I'm not saying you won't be able to do this in the future. However, you may need to cure a loathesome disease first.

Let's start with the essay. It's entitled "Gin, Television, and Social Surplus" by Clay Shirky, a teacher at New York University, a hotbed of advanced online thought. The essay appeared in 2008 and is well known among the cognoscenti, and my bringing it up now will strike some as on a par with getting excited about the Gettysburg Address. My apologies to these cutting-edge individuals. For the rest of you, here's the gist:

• In the early nineteenth century, the chief way people dealt with the stresses of factory work and urbanization was the consumption of  vast amounts of gin. Not till much later did institutions arise — Shirky cites museums, libraries and schools — that took advantage of urban aggregation to provide the citizenry with more constructive ways to occupy their time. 

• The equivalent of gin in the latter twentieth century was the TV sitcom, which soaked up the excess energy generated by increased postwar affluence and leisure: "Desperate Housewives essentially functioned as a kind of cognitive heat sink, dissipating thinking that might otherwise have built up and caused society to overheat," Shirky writes. In short, TV = opiate of the masses.

• The Web has at last provided us with an outlet for the mental energy heretofore sucked up by sitcoms. Exhibit A is Wikipedia, and one can only guess what additional marvels await.

The general formulation, then, is as follows: (a) massive societal upheaval produces social surplus; (b) lacking alternatives, we then blow off said surplus in useless dissipation (gin, TV); (c) eventually institutions and/or technology evolve to put the surplus to more productive use; (d) world becomes better place. To illustrate, Shirky calculates that Wikipedia to date has consumed 100 million hours of effort, which seems impressive until you consider that, in the U.S. alone, we spend 200 billion hours every year watching television. If we harnessed those squandered resources, Shirky observes (the common expression is "unused cycles"), we could produce the equivalent of 2,000 Wikipedias per year — and if that doesn't fill you with guilt, I don't know what will.

We've barely begun to plumb the subject, however. Let's move on.

Next on the reading list was The Wisdom of Crowds (2004). In it New Yorker business columnist James Surowiecki argues that groups of people, not all of whom are necessarily geniuses, often get better results than geniuses working alone. He cites some uncanny predictions, which I'd better describe in detail:

• In 1968 the U.S. submarine Scorpion vanished in the North Atlantic en route to its home port. The Navy knew little other than the vessel's last reported location. Rather than search randomly, naval officer John Craven drew up scenarios of the sub's fate — nature of failure, speed, rate of descent, and so on. Then he submitted the scenarios to experts who were asked to gauge the likelihood of each without consulting the others.  Finally, Craven performed statistically wizardry on the result to come up with a collective estimate of the sub's location. The sub, which might have been anywhere within a 20-mile circle, was found 220 yards from the estimated spot. Whoa. 

• British scientist Francis Galton in 1906 analyzed the results of a contest at a regional agricultural fair, in which participants were asked to guess the dressed weight (after butchering) of a certain ox. The contest was open to anyone, including farmers and ordinary jamokes. Eight hundred people gave it a shot. After the contest was over, Galton collected the entries and averaged the guesses: 1,197 pounds. The actual weight of the dressed ox: 1,198 pounds. Whoa again.

• On January 28, 1986, the space shuttle Challenger blew up shortly after launch. Investors promptly began dumping the stock of four contractors who'd help build it: Lockheed, Martin Marietta, Rockwell, and Morton Thiokol. (The last made the booster rockets, which featured something known as an O-ring.) All four stocks at first fell sharply. The first three partially recovered by the end of the day, but Morton Thiokol's kept falling and at the close of trading was down 12 percent, suggesting the market had determined the firm bore primary responsibility for the disaster. After painstaking investigation an official commission confirmed this conclusion six months later. The market had figured it out in one day.

By this time the reader is thoroughly freaked out. OK, the first two stories you can rationalize. Perusing the Wikipedia article about the Scorpion, we learn that underwater breakup noises helped the Navy get a fix on the vessel's location. Likewise, in the Galton story we may surmise that the collective wisdom of the knowledgeable farmers swamped the scattered efforts of everyone else.

Correctly assessing the blame for the Challenger disaster, on the other hand … there's a miracle for you. Surowiecki cites an analysis by two economists, who wind up conceding they can't explain it — there's no evidence anyone had inside information, and the traders had no scientific knowledge. So how did they do it — magic? ESP? Did they sense a disturbance in the force? No. Surowiecki explains:

At heart, the answer rests on a mathematical truism. If you ask a large enough group of diverse, independent people to make a prediction or estimate a probability, and then average those estimates, the errors each of them makes in coming up with an answer will cancel themselves out. Each person's guess, you might say, has two components: information and error. Subtract the error, and you're left with information.

I'm sure you'll join me in agreeing: this is deep.

Here we get down to the guts of the thing, in my opinion. However, some subtleties remain to be grasped. You might think the great big collective brain consists only of the brilliant. Not so. In the Web 2.0 view of things, a pivotal role is played by shlubs. Surowiecki writes:

Bringing new members into the organization, even if they're less experienced and less capable, actually makes the group smarter because what little the new members do know is not redundant with what everyone else knows [my emphasis].

He quotes organizational theorist James March:

The development of knowledge may depend on maintaining an influx of the naïve and the ignorant, and … competitive victory does not reliably go to the educated. 

Anybody who follows politics is thinking: no shit. However, there are broader implications. Surowiecki goes on to say:

What is striking … is just how much information a group's collective verdict so often contains …  the crowd is holding a nearly complete picture of the world in its collective brain.

There you have it. Let's review these key Web 2.0 concepts:

Follow STRAIGHT DOPE CHICAGO on twitter, assuming Little Ed remembers to send the feeds out, and if nothing else observe his fumblings with a technology that is obviously beyond him.

Send questions for Cecil Adams to cecil@chicagoreader.com or post them on the Straight Dope Chicago Message Board.

Send comments about this website to: webmaster@straightdope.com

Advertise on the Straight Dope! Your direct line to thousands of the smartest, hippest people on the planet, plus a few total dipsticks.

Copyright 1996-2008 Creative Loafing Media, Inc. All rights reserved. No material contained in this site may be republished or reposted without express written permission. The Straight Dope is a registered trademark of Creative Loafing Media, Inc.