Where Good Ideas Come From Part 2

A recent experiment led by the German neuroscientist Ullrich Wagner demonstrates the potential for dream states to trigger new conceptual insights. In Wagner's experiment, test subjects were a.s.signed a tedious mathematical task that involved the repet.i.tive transformation of eight digits into a different number. With practice, the test subjects grew steadily more efficient at completing the task. But Wagner's puzzle had a hidden pattern to it, a rule that governed the numerical transformations. Once discovered, the pattern allowed the subjects to complete the test much faster, not unlike the surge of activity one gets at the end of a jigsaw puzzle when all the pieces suddenly fall into place. Wagner found that after an initial exposure to the numerical test, "sleeping on the problem" more than doubled the test subjects' ability to discover the hidden rule. The mental recombinations of sleep helped them explore the full range of solutions to the puzzle, detecting patterns that they had failed to perceive in their initial training period. The work of dreams turns out to be a particularly chaotic, yet productive, way of exploring the adjacent possible.

In a sense, dreams are the mind's primordial soup: the medium that facilitates the serendipitous collisions of creative insight. And hunches are like those early carbon atoms, seeking out new kinds of connections to help them build new chains and rings of innovation. Loewi's dream about the frog heart experiment is often invoked as a story of sudden epiphany-a twentieth-century version of Newton's apple-but the truth is that Loewi had been musing on the idea that nerves might communicate chemically for seventeen years. In part, his epiphany was made possible by the random connections of REM sleep. Yet it was also made possible by a slow hunch that had been lingering in the back of his mind for almost two decades.

This pattern of a slow hunch crystallizing into a dream-inspired epiphany recurs in what may be the most famous reverie in the history of science. In 1865, the German chemist Friedrich August Kekule von Stradonitz had a daydream by a crackling fire in which he saw a vision of Ouroboros, the serpent from Greek mythology that devours its own tail. Kekule had spent the past ten years of his life exploring the connections of carbon-based molecules. The serpent image in his dream gave him a sudden insight into the molecular structure of the hydrocarbon benzene. The benzene molecule, he realized, was a perfect ring of carbon, with hydrogen atoms surrounding its outer edges. Kekule's slow hunch had set the stage for the insight, but for that hunch to turn into a world-changing idea, he needed the most unlikely of connections: an iconic image from ancient mythology. And Kekule's vision did indeed prove to be a breakthrough of epic proportions: the ring structure of the benzene molecule became the basis for a revolution in organic chemistry, opening up a new vista onto the mesmerizing array of rings, lattices, and chains formed by that most connective of elements, carbon. It took the combinatorial serendipity of a daydream-all those neurons firing in unlikely new configurations-to help us understand the combinatorial power of carbon, which was itself crucial to understanding the original innovations of life itself.

The waking brain, too, has an appet.i.te for the generative chaos that rules in the dream state. Neurons share information by pa.s.sing chemicals across the synaptic gap that connects them, but they also communicate via a more indirect channel: they synchronize their firing rates. For reasons that are not entirely understood, large cl.u.s.ters of neurons will regularly fire at the exact same frequency. (Imagine a discordant jazz band, each member following a different time signature and tempo, that suddenly snaps into a waltz at precisely 120 beats per minute.) This is what neuroscientists call phase-locking. There is a kind of beautiful synchrony to phase-locking-millions of neurons pulsing in perfect rhythm. But the brain also seems to require the opposite: regular periods of electrical chaos, where neurons are completely out of sync with each other. If you follow the various frequencies of brain-wave activity with an EEG, the effect is not unlike turning the dial on an AM radio: periods of structured, rhythmic patterns, interrupted by static and noise. The brain's systems are "tuned" for noise, but only in controlled bursts.

In 2007, Robert Thatcher, a brain scientist at the University of South Florida, decided to study the vacillation between phase-lock and noise in the brains of dozens of children. While Thatcher found that the noise periods lasted, on average, for 55 milliseconds, he also detected statistically significant variation among the children. Some brains had a tendency to remain longer in phase-lock, others had noise intervals that regularly approached 60 milliseconds. When Thatcher then compared the brain-wave results with the children's IQ scores, he found a direct correlation between the two data sets. Every extra millisecond spent in the chaotic mode added as much as twenty IQ points. Longer spells in phase-lock deducted deducted IQ points, though not as dramatically. IQ points, though not as dramatically.

Thatcher's study suggests a counterintuitive notion: the more disorganized your brain is, the smarter you are. It's counterintuitive in part because we tend to attribute the growing intelligence of the technology world with increasingly precise electromechanical ch.o.r.eography. Intel doesn't advertise its latest microprocessors with the slogan: "Every 55 milliseconds, our chips erupt into a blizzard of noise!" Yet somehow brains that seek out that noise seem to thrive, at least by the measure of the IQ test.

Science does not yet have a solid explanation for the brain's chaos states, but Thatcher and other researchers believe that the electric noise of the chaos mode allows the brain to experiment with new links between neurons that would otherwise fail to connect in more orderly settings. The phase-lock mode (the theory goes) is where the brain executes an established plan or habit. The chaos mode is where the brain a.s.similates new information, explores strategies for responding to a changed situation. In this sense, the chaos mode is a kind of background dreaming: a wash of noise that makes new connections possible. Even in our waking hours, it turns out, our brains gravitate toward the noise and chaos of dreams, 55 milliseconds at a time.

William James, writing in the late 1880s, had no way of measuring synchronized neuron firing, but his description of the "highest order of minds" captures something of the chaos mode: Instead of thoughts of concrete things patiently following one another, we have the most abrupt cross-cuts and transitions from one idea to another, the most rarefied abstractions and discriminations, the most unheard-of combinations of elements . . . a seething caldron of ideas, where everything is fizzling and bobbing about in a state of bewildering activity, where partners.h.i.+ps can be joined or loosened in an instant, treadmill routine is unknown, and the unexpected seems the only law.

The act of s.e.xual reproduction is itself a kind of testament to the power of random connections, even in the most monogamous relations.h.i.+ps. The overwhelming majority of nonmicroscopic life on earth produces offspring by sharing genes with another organism. But the evolution of this reproductive strategy remains something of a mystery. It would have been far easier for life to have avoided the complicated genetic exchanges of meiosis and fertilization. (Think of the elaborate system that the flowering plants had to evolve, luring insects to take on the task of carrying pollen from flower to flower.) Reproduction without s.e.x is a simple matter of cloning; you take your own cells, make a copy, and pa.s.s that on to your descendants. It doesn't sound like much fun to our mammalian ears, but it's a strategy that has worked very well for billions of years for bacteria. As.e.xual reproduction is faster and more energy efficient than the s.e.xual variety: you don't need to go to the trouble of finding a partner in order to create the next generation.

If natural selection rewarded organisms exclusively for sheer reproductive power, s.e.xual reproduction might never have evolved. As.e.xual organisms reproduce on average twice as quickly as their s.e.xual counterparts, in part because without a male/female distinction, every organism is capable of producing offspring directly. But evolution is not just a game of sheer quant.i.ty. Overpopulation, after all, poses its own dangers, and a community of organisms with identical DNA makes a prime target for parasites or predators. For these reasons, natural selection also rewards innovation, life's tendency to discover new ecological niches, new sources of energy. This is what Stuart Kauffman recognized when he first formulated the idea of the adjacent possible: that there is something like an essential drive in the biosphere to diversify into new ways of making a living. Scrambling together two distinct sets of DNA with each generation made for a far more complicated reproductive strategy, but it paid immense dividends in the rate of innovation. What we gave up in speed and simplicity, we made up for in creativity.

The water flea Daphnia lives in most freshwater ponds and swamps. Its spasmodic movements in the water are responsible for the "flea" description, but in reality Daphnia are tiny crustaceans, no more than a few millimeters long. Under normal conditions, Daphnia reproduce as.e.xually, with females producing a brood of identical copies of themselves in a tiny pouch. In this mode, the Daphnia community is composed entirely of females. This reproductive strategy proves to be stunningly successful: in warm summer months, Daphnia will often be one of the most abundant organisms in a pond ecosystem. But when conditions get tough, when droughts or other ecological disturbances happen, or when winter rolls in, the water fleas make a remarkable transformation: they start producing males and switch to reproducing s.e.xually. In part, this switch is attributable to the st.u.r.dier eggs produced by s.e.xual reproduction, which are more capable of surviving the long months of winter. But scientists believe that the sudden adoption of s.e.x is also a kind of biological innovation strategy: in challenging times, an organism needs new ideas to meet those new challenges. Reproducing as.e.xually makes perfect sense during prosperous periods: if life is good, keep doing what you're doing. Don't mess with success by introducing new genetic combinations. But when the world gets more challenging-scarce resources, predators, parasites-you need to innovate. And the quickest path to innovation lies in making novel connections. This strategy of switching back and forth between as.e.xual and s.e.xual reproduction goes by the name "heterogamy," and while it is unusual, many different organisms have adopted it. Slime molds, algae, and aphids have all evolved heterogamous reproductive strategies. In each organism, the Daphnia pattern repeats itself: the genetic recombinations of s.e.x emerge when conditions get difficult. Swapping genes with another organism is itself more difficult than simple cloning, but the innovation rewards of s.e.x outweigh the risks of the more stable path. When nature finds itself in need of new ideas, it strives to connect, not protect.

The English language is blessed with a wonderful word that captures the power of accidental connection: "serendipity." First coined in a letter written by the English novelist Horace Walpole in 1754, the word derives from a Persian fairy tale t.i.tled "The Three Princes of Serendip," the protagonists of which were "always making discoveries, by accident and sagacity, of things they were not in quest of." The contemporary novelist John Barth describes it in nautical terms: "You don't reach Serendip by plotting a course for it. You have to set out in good faith for elsewhere and lose your bearings serendipitously."

But serendipity is not just about embracing random encounters for the sheer exhilaration of it. Serendipity is built out of happy accidents, to be sure, but what makes them happy is the fact that the discovery you've made is meaningful to you. It completes a hunch, or opens up a door in the adjacent possible that you had overlooked. If you're a geologist randomly exploring the Web, and the particular isle of Serendip that you stumble across turns out to be an essay on health-care reform, your discovery may well be interesting and informative, but it will not be truly serendipitous unless it helps you fill in a piece of a puzzle you've been poring over. That's not to say that geologists can only find serendipitous discoveries in texts about geology-quite the contrary, in fact. Serendipitous discoveries often involve exchanges across traditional disciplines. Think of the way Kekule's mythic serpent led to a revolution in organic chemistry. It was genuinely serendipitous that Kekule's dreaming brain should conjure up the image of Ouroboros at that moment. But had Kekule not been wrestling with the structure of the benzene molecule for years, that serpent shape might not have triggered any useful a.s.sociations in his mind. (Sometimes a serpent swallowing its tail is just a serpent swallowing its tail, as Freud might have said.) Serendipity needs unlikely collisions and discoveries, but it also needs something to anchor those discoveries. Otherwise, your ideas are like carbon atoms randomly colliding with other atoms in the primordial soup without ever forming the rings and lattices of organic life.

The challenge, of course, is how to create environments that foster these serendipitous connections, on all the appropriate scales: in the private s.p.a.ce of your own mind; within larger inst.i.tutions; and across the information networks of society itself.

At first blush, the idea of conjuring up serendipitous discoveries inside your own mind seems like a contradiction in terms. Wouldn't that be like losing your bearings in your own driveway? Yet that's exactly what Kekule was doing by the fire. He was connecting two distinct thoughts that each occupied a slot in his memory banks: the riddle of benzene's molecular structure, and the tail-swallowing Ouroboros. The truth is, your mind contains a near-infinite number of ideas and memories that at any given moment are lurking outside your consciousness. Some tiny fraction of those thoughts are like Kekule's serpent: surprising connections that might help you unlock a door in the adjacent possible. But how do you get those particular cl.u.s.ters of neurons to fire at the right time?

One way is to go for a walk. The history of innovation is replete with stories of good ideas that occurred to people while they were out on a stroll. (A similar phenomenon occurs with long showers or soaks in a tub; in fact, the original "eureka" moment-Archimedes. .h.i.tting upon a way of measuring the volume of irregular shapes-occurred in a bathtub.) The shower or stroll removes you from the task-based focus of modern life-paying bills, answering e-mail, helping kids with homework-and deposits you in a more a.s.sociative state. Given enough time, your mind will often stumble across some old connection that it had long overlooked, and you experience that delightful feeling of private serendipity: Why didn't I think of that before?

In his book The Foundations of Science The Foundations of Science, the French mathematician and physicist Henri Poincare devotes an autobiographical chapter to the question of mathematical creativity. The chapter begins with a detailed account of how Poincare discovered the cla.s.s of Fuchsian functions, one of the first influential mathematical concepts of his career. He begins by attempting to prove that the functions do not exist; for fifteen days he struggles at his desk with no success. Then one evening he breaks from his ordinary routine and drinks black coffee. Unable to sleep, his mind seethes with promising hunches. "Ideas rose in crowds," Poincare writes. "I felt them collide until pairs interlocked, so to speak, making a stable combination. By the next morning I had established the existence of a cla.s.s of Fuchsian functions, those which come from the hypergeometric series." His next insight-a connection between the functions and non-Euclidean geometry-comes several weeks later, while boarding a bus during a geological expedition in Normandy. On his return home, he commences work on an unrelated arithmetical question and flounders for several days. "Disgusted with my failure," he writes, "I went to spend a few days at the seaside, and thought of something else. One morning, walking on the bluff, the idea came to me, with just the same characteristics of brevity, suddenness and immediate certainty, that the arithmetic transformations of indeterminate ternary quadratic forms were identical with those of non-Euclidean geometry." He returns home again and works through the implications, but encounters another roadblock. Military service then dictates a trip to Fort Mont-Valerien in the suburbs of Paris, where he has little time to think about mathematics at all. And yet the final missing piece arrives nonetheless. "One day, going along the street, the solution of the difficulty which had stopped me suddenly appeared to me. I did not try to go deep into it immediately, and only after my service did I again take up the question. I had all the elements and had only to arrange them and put them together. So I wrote out my final memoir at a single stroke and without difficulty."

Poincare's account may be the most "pedestrian" story of scientific creativity on record. Whenever he actually sits down at his desk, the innovations seem to grind to a halt. But on foot, his ideas "rose in crowds." Trying to explain the phenomenon, Poincare reaches for an atomic metaphor, with each partial idea or hunch represented by an atom hooked to a wall. In normal situations, the atoms remain in place, locked into a stable configuration. But when the mind wanders (and, in Poincare's case, when the physical body wanders), the atoms become untethered. "During a period of apparent rest and unconscious work, certain of them are detached from the wall and put in motion. They flash in every direction through the s.p.a.ce . . . where they are enclosed, as would, for example, a swarm of gnats or, if you prefer a more learned comparison, like the molecules of gas in the kinematic theory of gases. Then their mutual impacts may produce new combinations."

While the creative walk can produce new serendipitous combinations of existing ideas in our heads, we can also cultivate serendipity in the way that we absorb new ideas from the outside world. Reading remains an unsurpa.s.sed vehicle for the transmission of interesting new ideas and perspectives. But those of us who aren't scholars or involved in the publis.h.i.+ng business are only able to block out time to read around the edges of our work schedule: listening to an audio book during the morning commute, or taking in a chapter after the kids are down. The problem with a.s.similating new ideas at the fringes of your daily routine is that the potential combinations are limited by the reach of your memory. If it takes you two weeks to finish a book, by the time you get to the next book, you've forgotten much of what was so interesting or provocative about the original one. You can immerse yourself in a single author's perspective, but then it's harder to create serendipitous collisions between the ideas of multiple authors. One way around this limitation is to carve out dedicated periods where you read a large and varied collection of books and essays in a condensed amount of time. Bill Gates (and his successor at Microsoft, Ray Ozzie) are famous for taking annual reading vacations. During the year they deliberately cultivate a stack of reading material-much of it unrelated to their day-to-day focus at Microsoft-and then they take off for a week or two and do a deep dive into the words they've stockpiled. By compressing their intake into a matter of days, they give new ideas additional opportunities to network among themselves, for the simple reason that it's easier to remember something that you read yesterday than it is to remember something you read six months ago.

In Poincare's language, the deep dive, like the long stroll, detaches the atoms from the wall and puts them in motion. Most of us don't have the luxury of taking deep dive reading sabbaticals, of course, and reading a few thousand pages is not everyone's idea of a fun vacation. But there's no reason why organizations couldn't recognize the value of a reading sabbatical, the way many organizations encourage their employees to take time off for learning new skills. If Google can give its engineers one day a week to work on anything they want, surely other organizations can figure out a way to give their employees dedicated time to immerse themselves in a network of new ideas.

Private serendipity can be cultivated by technology as well. For more than a decade now, I have been curating a private digital archive of quotes that I've found intriguing, my twenty-first-century version of the commonplace book. Some of these pa.s.sages involve very focused research on a specific project; others are more random discoveries, hunches waiting to make a connection. Some of them are pa.s.sages that I've transcribed from books or articles; others were clipped directly from Web pages. (In the past few years, thanks to Google Books and the Kindle, copying and storing interesting quotes from a book has grown far simpler.) I keep all these quotes in a database using a program called DEVONthink, where I also store my own writing: chapters, essays, blog posts, notes. By combining my own words with pa.s.sages from other sources, the collection becomes something more than just a file storage system. It becomes a digital extension of my imperfect memory, an archive of all my old ideas, and the ideas that have influenced me. There are now more than five thousand distinct entries in that database, and more than 3 million words-sixty books' worth of quotes, fragments, and hunches, all individually captured by me, stored in a single database.

Having all that information available at my fingertips is not just a quant.i.tative matter of finding my notes faster. Yes, when I'm trying to track down an article I wrote many years ago, it's now much easier to retrieve. But the qualitative change lies elsewhere: in finding doc.u.ments that I've forgotten about altogether, finding doc.u.ments that I didn't know I was looking for. What makes the system truly powerful is the way that it fosters private serendipity.

DEVONthink features a clever algorithm that detects subtle semantic connections between distinct pa.s.sages of text. These tools are smart enough to get around the cla.s.sic search-engine failing of excessive specificity: searching for "dog" and missing all the articles that only have the word "canine" in them. Modern indexing software like DEVONthink's learns a.s.sociations between individual words by tracking the frequency with which words appear near each other. This can create almost lyrical connections between ideas. Several years ago, I was working on a book about cholera in London and queried DEVONthink for information about Victorian sewage systems. Because the software had detected that the word "waste" is often used alongside "sewage," it directed me to a quote that explained the way bones evolved in vertebrate bodies: namely, by repurposing the calcium waste products created by the metabolism of cells. At first glance that might seem like an errant result, but it sent me off on a long and fruitful tangent into the way complex systems-whether cities or bodies-find productive uses for the waste they create. That idea became a central organizing theme for one of the chapters in the cholera book. (It will, in fact, reappear in this book in a different guise.) Now, strictly speaking, who was responsible for that initial idea? Was it me, or the software? It sounds like a facetious question, but I mean it seriously. Obviously, the computer wasn't conscious conscious of the idea taking shape, and I supplied the conceptual glue that linked the London sewers to cell metabolism. But I'm not at all confident that I would have made the initial connection without the help of the software. The idea was a true collaboration, two very different kinds of intelligence playing off one another, one carbon-based, the other silicon. When I'd first captured that quote about calcium and bone structure, I'd had no idea that it would ultimately connect to the history of London's sewage system (or to a book about innovation). But there was something about that concept that intrigued me enough to store it in the database. It lingered there for years in the software's primordial soup, a slow hunch waiting for its connection. of the idea taking shape, and I supplied the conceptual glue that linked the London sewers to cell metabolism. But I'm not at all confident that I would have made the initial connection without the help of the software. The idea was a true collaboration, two very different kinds of intelligence playing off one another, one carbon-based, the other silicon. When I'd first captured that quote about calcium and bone structure, I'd had no idea that it would ultimately connect to the history of London's sewage system (or to a book about innovation). But there was something about that concept that intrigued me enough to store it in the database. It lingered there for years in the software's primordial soup, a slow hunch waiting for its connection.

I use DEVONthink as an improvisational tool as well. I write a paragraph about something-let's say it's about the human brain's remarkable facility for interpreting facial expressions. I then plug that paragraph into the software, and ask DEVONthink to find other pa.s.sages in my archive that are similar. Instantly, a list of quotes appears on my screen: some delving into the neural architecture that triggers facial expressions, others exploring the evolutionary history of the smile, others dealing with the expressiveness of our near-relatives, the chimpanzees. Invariably, one or two of these triggers a new a.s.sociation in my head-perhaps I've forgotten about the chimpanzee connection-and so I select that quote, and ask the software to find a new batch of pa.s.sages similar to it. Before long, a larger idea takes shape in my head, built upon the trail of a.s.sociations the machine has a.s.sembled for me.

Compare that to the traditional way of exploring your files, where the computer is like a dutiful, but dumb, butler: "Find me that doc.u.ment about the chimpanzees!" That's searching. The other feels radically different, so different that we don't quite have a verb for it: it's riffing, or exploring. There are false starts and red herrings, but there are just as many happy accidents and unexpected discoveries. Indeed, the fuzziness of the results is part of what makes the software so powerful. The serendipity of the system emerges out of two distinct forces. First, there is the connective power of the semantic algorithm, which is smart but also slightly unpredictable, thus creating a small amount of randomizing noise that makes the results more surprising. But that randomizing force is held in check by the fact that I have curated all these pa.s.sages myself, which makes each individual connection far more likely to be useful to me in some way. When you start a new query in DEVONthink and look down at the initial results, at first glance they can sometimes seem jumbled and disconnected, but then you read through them in more detail, and inevitably something tantalizing catches your eye. "Jumbled" and "disconnected" is of course also how we describe the strange explorations of our dreams, and the comparison is an apt one. DEVONthink takes the strange but generative combinations of the dream state and turns them into software.

If you visit the "serendipity" entry in Wikipedia, you are one click away from entries on LSD, Teflon, Parkinson's disease, Sri Lanka, Isaac Newton, v.i.a.g.r.a, and about two hundred other topics of comparable diversity. That eclecticism is particularly acute at Wikipedia, of course, but it derives from the fundamentally "tangled" nature of Tim Berners-Lee's original hypertext architecture. No medium in history has ever offered such unlikely trails of connection and chance in such an intuitive and accessible form. Yet in recent years, a puzzling meme has emerged on op-ed pages with a strange insistence: the rise of the Web, its proponents argue, has led to a decline decline in serendipitous discovery. Consider this representative elegy to the "endangered joy of serendipity," auth.o.r.ed by a journalism professor named William McKeen: in serendipitous discovery. Consider this representative elegy to the "endangered joy of serendipity," auth.o.r.ed by a journalism professor named William McKeen: Think about the library. Do people browse anymore? We have become such a directed people. We can target what we want, thanks to the Internet. Put a couple of key words into a search engine and you find-with an irritating hit or miss here and there-exactly what you're looking for. It's efficient, but dull. You miss the time-consuming but enriching act of looking through shelves, of pulling down a book because the t.i.tle interests you, or the binding . . . Looking for something and being surprised by what you find-even if it's not what you set out looking for-is one of life's great pleasures, and so far no software exists that can duplicate that experience.

In a similar piece, the New York Times New York Times technology editor, Damon Darlin, complained that the "digital age is stamping out serendipity." Darlin acknowledged the vast influx of suggested reading that now arrives on our screen every morning via social network services like Twitter and Facebook, but claimed those links didn't const.i.tute serendipity. "[They're] really group-think," Darlin argued. "Everything we need to know comes filtered and vetted. We are discovering what everyone else is learning, and usually from people we have selected because they share our tastes." technology editor, Damon Darlin, complained that the "digital age is stamping out serendipity." Darlin acknowledged the vast influx of suggested reading that now arrives on our screen every morning via social network services like Twitter and Facebook, but claimed those links didn't const.i.tute serendipity. "[They're] really group-think," Darlin argued. "Everything we need to know comes filtered and vetted. We are discovering what everyone else is learning, and usually from people we have selected because they share our tastes."

When critics complain about the decline of serendipity, they habitually point to two "old media" mechanisms that allegedly have no direct equivalent on the Web. McKeen mentions the first one: browsing the stacks in a library (or a bookstore), "pulling down a book because the t.i.tle interests you, or the binding." Old-style browsing does indeed lead to unplanned discoveries. But thanks to the connective nature of hypertext, and the blogosphere's exploratory hunger for finding new stuff, it is far easier to sit down in front of your browser and stumble across something completely brilliant but surprising than it is to walk through a library, looking at the spines of books. Does everyone use the Web this way? Of course not. But it is much more of a mainstream pursuit than randomly exploring the library stacks, pulling down books because you like the binding, ever was. This is the irony of the serendipity debate: the thing that is being mourned has actually gone from a fringe experience to the mainstream of the culture.

The second a.n.a.log-era mechanism that encourages serendipity involves the physical limitations of the print newspaper, which forces you to pa.s.s by a collection of artfully curated stories on a variety of topics, before you open up the section that most closely matches your existing pa.s.sions and knowledge. The legal scholar Ca.s.s Sunstein refers to this as an example of the "architecture of serendipity." On the way to the sports section or the comics or the business page, you happen to collide with a story about the abuses of African diamond mines, and something in the headline catches your eye. A thousand words later, you've learned something powerful about people living halfway around the world whose existence you had never contemplated before. And perhaps there is some kind of serendipitous click in that collision: you'd been looking for a new charitable cause to support, or contemplating buying your spouse a diamond ring. And then this story drops in your lap, and helps you complete the thought. You weren't looking for a story about diamond mines, but it was exactly what you needed.

This is indeed a superb example of serendipity, and there is no doubt that newspapers facilitated comparable accidental discoveries countless times over countless breakfast tables during their heyday. The question is whether the transition to the Web makes this sort of discovery more or less frequent. If you compare the front pages of the print and online versions of a newspaper, the Web actually appears to have the upper hand. The Internet scholar Ethan Zuckerman compared the front page of the New York Times New York Times with that of its Web cousin and found that the print version had twenty-three references on its front page to articles in the paper (either in the form of lead articles themselves, or short summaries teased below the fold). The front page at with that of its Web cousin and found that the print version had twenty-three references on its front page to articles in the paper (either in the form of lead articles themselves, or short summaries teased below the fold). The front page at NYTimes.com, in Zuckerman's study, contained 315 links to articles and other forms of content. If the architecture of serendipity lies in stumbling across surprising connections while scanning the front page, then the Web is more than ten times as serendipitous as the cla.s.sic print newspaper.

Sunstein would no doubt argue that many people bypa.s.s the front door of their online newspaper, going directly to the sports or business-section page that they've bookmarked, or to some other filter tailor-made to their preexisting interests. No doubt millions of people use comparable filters every morning. One could reasonably question whether people like this who have gone out of their way to avoid encountering the "big picture" of the newspaper front page were ever likely to stumble across the diamond-mining story at the breakfast table with a print paper, or ramble through the stacks of their local library. But Sunstein and Darlin and McKeen are indeed correct when they argue that the Internet gives us topical filters that were unheard of in the days of ma.s.s media. But those filters are only part of the story. Filters reduce serendipity (unless your particular interest lies in being surprised, which is part of the appeal of beautifully miscellaneous blogs like Boing Boing). But beyond bookmarking, filters are a second-generation addition to the architecture of the Web. They are not native to it. What is is native to the Web's architecture are two key features that have been great supporters of serendipity: a global, distributed medium in which anyone can be a publisher, and a hypertext doc.u.ment structure in which it is trivial to jump from a newspaper article to an academic essay to an encyclopedia entry in a matter of seconds. The information diversity of the Web ensures that there is an endless supply of surprising information to stumble across, and the links of hypertext ensure that we can get to that information at lightning speed, or follow trails of improvised a.s.sociation that would have been painfully slow to follow in the age of print media. Ironically, the problem with the Web is that there's too much noise, too much chaos-that's why the filters were invented in the first place. We have filters because the Web has unleashed too much diversity and surprise, not because we have too little. native to the Web's architecture are two key features that have been great supporters of serendipity: a global, distributed medium in which anyone can be a publisher, and a hypertext doc.u.ment structure in which it is trivial to jump from a newspaper article to an academic essay to an encyclopedia entry in a matter of seconds. The information diversity of the Web ensures that there is an endless supply of surprising information to stumble across, and the links of hypertext ensure that we can get to that information at lightning speed, or follow trails of improvised a.s.sociation that would have been painfully slow to follow in the age of print media. Ironically, the problem with the Web is that there's too much noise, too much chaos-that's why the filters were invented in the first place. We have filters because the Web has unleashed too much diversity and surprise, not because we have too little.

I happen to believe that the Web, as a medium, has pushed the culture toward more serendipitious encounters. The simple fact that information "browsing" and "surfing" are now mainstream pursuits makes a strong case for a rise in serendipity, compared to cultures dominated by books or ma.s.s media. But whether or not you accept the premise that the average media consumer experiences more serendipitous discoveries thanks to the Web, there can be little doubt that the Web is an unrivaled medium for serendipity if you are actively seeking it out. If you want to build a daily reading list of eclectic and diverse perspectives, you can st.i.tch one together in your RSS reader or your bookmarks bar in a matter of minutes, for no cost, while sitting on your couch. Just as important, you can use the Web to fill out the context when you do stumble across some interesting new topic. The great oracle of the digital age, Google, is often invoked as a serendipity killer, because search queries function as a kind of on-demand filter that eliminates the 99.999 percent of the Web that is not relevant to the searcher's current interest. But when critics put Google on the side of filters, they a.s.sume that most queries are variations on the theme: "I'm pa.s.sionately interested in x x and would like to learn more about it." No doubt some unthinkably large number of Google users enter queries that take that basic shape every day. But there's another type of query that is just as valuable: "Someone just told me about and would like to learn more about it." No doubt some unthinkably large number of Google users enter queries that take that basic shape every day. But there's another type of query that is just as valuable: "Someone just told me about x x and I know nothing about it, but it sounds interesting. Tell me more." This is the subtle way in which Google supports the serendipitous aspects of the Web. Yes, it's true that by the time you've entered something into the Google search box, you're already invested in the topic. (This is why Web pioneer John Battelle calls it the "database of intentions.") But often that investment is directly correlated with your ignorance about the topic at hand: someone mentions in pa.s.sing the poetry of John Ashbery, or the television show and I know nothing about it, but it sounds interesting. Tell me more." This is the subtle way in which Google supports the serendipitous aspects of the Web. Yes, it's true that by the time you've entered something into the Google search box, you're already invested in the topic. (This is why Web pioneer John Battelle calls it the "database of intentions.") But often that investment is directly correlated with your ignorance about the topic at hand: someone mentions in pa.s.sing the poetry of John Ashbery, or the television show Arrested Development Arrested Development, or the tail-swallowing Ouroboros, and you think: "What's the deal with that? It sounds really interesting." Imagine it's 1980 and you're sitting at your breakfast table, reading the morning paper, and on the way to the sports page you stumble across an article on the front page about this provocative new idea of global warming that you've not yet encountered. You can read the article, to be sure, but when the article leaves you hankering for more information and context, where do you go? Turn on the television and hope that one of the three networks or PBS is running a news item or a doc.u.mentary on the topic at that exact second? Get in the car, drive fifteen minutes down to your public library and check out a book on the subject? Go through all the magazines in your house, scouring their table-of-contents pages for any climate-change-related articles?

Let's say you live in a particularly information-rich household for the standards of 1980, and you happen to have a copy of the Encyclopoedia Britannica Encyclopoedia Britannica. But of course the version you bought is actually the 1976 edition, and global warming doesn't make it into the Britannica Britannica until 1994, despite the fact that the term is common enough to be mentioned in ordinary parlance throughout the nineties. until 1994, despite the fact that the term is common enough to be mentioned in ordinary parlance throughout the nineties.

Today, of course, you would query either Google or Wikipedia for the search term "global warming." And you would instantly have more information (and more perspectives) at your fingertips than would have been imaginable when you were thumbing through the Britannica Britannica in 1980. Yes, these results are targeted to your expressed interest in a specific topic, but that interest is often something you've just stumbled across, a hint more than a pa.s.sion. And because those pages are built out of hyperlinks, just a few clicks can land you in an entirely new region of interest that you'd never dreamed of visiting. Google and Wikipedia give those pa.s.sing hints something to attach to, a kind of information anchor that lets you settle down around a topic and explore the surrounding area. They turn hints and happy accidents into information. If the commonplace book tradition tells us that the best way to nurture hunches is to write everything down, the serendipity engine of the Web suggests a parallel directive: in 1980. Yes, these results are targeted to your expressed interest in a specific topic, but that interest is often something you've just stumbled across, a hint more than a pa.s.sion. And because those pages are built out of hyperlinks, just a few clicks can land you in an entirely new region of interest that you'd never dreamed of visiting. Google and Wikipedia give those pa.s.sing hints something to attach to, a kind of information anchor that lets you settle down around a topic and explore the surrounding area. They turn hints and happy accidents into information. If the commonplace book tradition tells us that the best way to nurture hunches is to write everything down, the serendipity engine of the Web suggests a parallel directive: look everything up look everything up.

The premise that innovation prospers when ideas can serendipitously connect and recombine with other ideas, when hunches can stumble across other hunches that successfully fill in their blanks, may seem like an obvious truth, but the strange fact is that a great deal of the past two centuries of legal and folk wisdom about innovation has pursued the exact opposite argument, building walls between ideas, keeping them from the kind of random, serendipitous connections that exist in dreams and in the organic compounds of life. Ironically, those walls have been erected with the explicit aim of encouraging innovation. They go by many names: patents, digital rights management, intellectual property, trade secrets, proprietary technology. But they share a founding a.s.sumption: that in the long run, innovation will increase if you put restrictions on the spread of new ideas, because those restrictions will allow the creators to collect large financial rewards from their inventions. And those rewards will then attract other innovators to follow in their path.4 The problem with these closed environments is that they inhibit serendipity and reduce the overall network of minds that can potentially engage with a problem. This is why a growing number of large organizations-businesses, nonprofits, schools, government agencies-have begun experimenting with work environments that encourage the architecture of serendipity. Traditionally, organizations that have a strong demand for innovation have created a kind of closed playpen for hunches: the research-and-development lab. Ironically, R&D labs have historically functioned as a kind of idea lockbox; the hunches evolving in those labs tended to be the most heavily guarded secrets in the entire organization. Allowing these early product ideas to circulate more widely would allow rival firms to copy or exploit them. Some organizations-including Apple-have gone to great length to keep R&D experiments sequestered from other employees inside inside the organization. the organization.

But that secrecy, as we have seen, comes with great cost. Protecting ideas from copycats and compet.i.tors also protects them from other ideas that might improve them, might transform them from hints and hunches to true innovations. And indeed there is a grow-ing movement in some forward-thinking companies to turn their R&D labs inside out and make them far more transparent than the traditional model. Organizations like IBM and Procter & Gamble, who have a long history of profiting from patented, closed-door innovations, have embraced open innovation platforms over the past decade, sharing their leading-edge research with universities, partners, suppliers, and customers.

In early 2010, Nike announced a new Web-based marketplace it called the GreenXchange, where it publicly released more than 400 of its patents that involve environmentally friendly materials or technologies. The marketplace was a kind of hybrid of commercial self-interest and civic good. By making its good ideas public, Nike made it possible for outside firms to improve on those innovations, creating new value that Nike itself might ultimately be able to put to use in its own products. In a sense, Nike was widening the network of minds who were actively thinking about how to make its ideas more useful, without putting anyone else on its payroll. But Nike's organizational values also include a commitment to environmental sustainability, and the company recognized that many of its eco-friendly patents might be useful in different contexts. Nike is a big corporation, with many products in many categories, but there are limits to its reach. Some of its innovations might well turn out to be advantageous to industries or markets where it has no compet.i.tive involvement whatsoever. By keeping its eco-friendly ideas behind a veil of secrecy, Nike was holding back-without any real commercial justification-ideas that might, in another context, contribute to a sustainable future. In collaboration with Creative Commons, Nike released its patents under a modified license permitting use in "non-compet.i.tive" fields. (They also created a standardized, pre-negotiated contract for the patents, thereby reducing the transaction costs of haggling over each patent license individually.) The example scenario they invoked at the launch of GreenXchange would have warmed the heart of Stephen Jay Gould: an environmentally sound rubber originally invented for use in running shoes that could be adapted by a mountain bike company to create more sustainable tires. Apparently, Gould's tires-to-sandals principle works both ways. Sometimes you make footwear by putting tires to new use, sometimes you make tires by putting footwear to new use. Green Xchange is trying to give multinational corporations some of the same freedom to reinvent and recycle that Gould's sandal-makers enjoy sifting through the Nairobi junkyards.

The other organizational technique for facilitating serendipitous connections is the "brainstorm" session, an approach pioneered by the advertising executive Alex Osborn in the 1930s. Brainstorming opens up the flow of ideas and hunches in a more generative fas.h.i.+on than is customary in a regimented workplace meeting. Yet a number of recent studies have suggested that brainstorming is less effective than its pract.i.tioners would like. One trouble with brainstorming is that it is finite in both time and s.p.a.ce: a group gathers for an hour in a room, or for a daylong corporate retreat, they toss out a bunch of crazy ideas, and then the meeting disperses. Sometimes a useful connection emerges, but too often the relevant hunches aren't in sync with one another. One employee has a promising hunch in one office, and two months later, another employee comes up with the missing piece that turns that hunch into a genuine insight. Brainstorming might bring those two fragments together, but the odds are against it. Imagine some kind of alternate reality where the FBI holds a corporate retreat in late August of 2001, and invites the field agents from Arizona and Minnesota to sit in a room together and brainstorm new potential threats against the United States. No doubt it would have been the first corporate retreat on record that actually changed the fate of world history, but with more than ten thousand field agents across the nation, the odds against getting the right people from Arizona and Minnesota together at the right time would have been astronomical. But imagine if the FBI had been using a networked version of a DEVONthink archive instead of the archaic Automated Case Support system. The top bra.s.s at the Radical Fundamentalist Unit would still have read the search warrant request for Moussaoui's laptop and thought to themselves, "This sounds like a pretty shaky hunch." But a quick DEVONthink query would have pointed them to the Phoenix memo, to another hunch about flight training and terrorism. Those two unlikely ideas would have collided, without the field agents in Phoenix and Minnesota even speaking to each other, much less sitting down for a brainstorming session.

The secret to organizational inspiration is to build information networks that allow hunches to persist and disperse and recombine. Instead of cloistering your hunches in brainstorm sessions or R&D labs, create an environment where brainstorming is something that is constantly running in the background, throughout the organization, a collective version of the 20-percent-time concept that proved so successful for Google and 3M. One way to do this is to create an open database of hunches, the Web 2.0 version of the traditional suggestion box. A public hunch database makes every pa.s.sing idea visible to everyone else in the organization, not just management. Other employees can comment or expand on those ideas, connecting them with their own hunches about new products or priorities or internal organizational changes. Some systems even allow employees to vote on their colleagues' suggestions, not unlike the user rankings that power collective news sites like Digg or Reddit. Google has a company-wide e-mail list where employees can suggest new features or products; each suggestion can then be rated on a scale of 0 ("Dangerous or harmful") to 5 ("Great idea! Make it so."). Salesforce.com maintains a popular Idea Exchange where its customers can suggest new features for the company's software products. The Idea Exchange doesn't just allow interesting hunches to circulate and connect. It also tracks their maturation into s.h.i.+pping code: the front door of the Exchange includes prominent links to submitted ideas currently being considered for inclusion in future releases, as well as ideas that were successfully integrated into past releases. Too often, real-world suggestion boxes feel like a black hole; you drop your idea in the slot, and never hear about it again. In a public forum like Idea Exchange, not only do you get to see and improve other people's suggestions, but you get tangible evidence that your ideas can make a difference. maintains a popular Idea Exchange where its customers can suggest new features for the company's software products. The Idea Exchange doesn't just allow interesting hunches to circulate and connect. It also tracks their maturation into s.h.i.+pping code: the front door of the Exchange includes prominent links to submitted ideas currently being considered for inclusion in future releases, as well as ideas that were successfully integrated into past releases. Too often, real-world suggestion boxes feel like a black hole; you drop your idea in the slot, and never hear about it again. In a public forum like Idea Exchange, not only do you get to see and improve other people's suggestions, but you get tangible evidence that your ideas can make a difference.

These kinds of information networks can do a masterful job of tapping both individual and collective intelligence: the individual employee has a provocative and useful hunch, and the group helps complete the hunch by connecting it to other ideas that have circulated through the system, and helps separate out that hunch from the thousands of other less useful ones by voting it to the top of the charts. By making the ideas public, and by ensuring that they remain stored in the database, these systems create an architecture for organizational serendipity. They give good ideas new ways to connect.

V.

ERROR.

In the summer of 1900 a twenty-seven-year-old aspiring inventor named Lee de Forest moved to Chicago, rented a one-room apartment on Was.h.i.+ngton Boulevard, and took a day job translating foreign articles on wireless technology for Western Electrician Western Electrician magazine. The translation work was informative: a major exposition on wireless technology that had just been held in Paris guaranteed a constant flow of interesting new research papers across the Atlantic. But de Forest's true pa.s.sion lay in the cabinet of wonders he had a.s.sembled in his bedroom on Was.h.i.+ngton Boulevard: batteries, spark gap transmitters, electrodes-all the building blocks that would be a.s.sembled in the coming decade to invent the age of electronics. magazine. The translation work was informative: a major exposition on wireless technology that had just been held in Paris guaranteed a constant flow of interesting new research papers across the Atlantic. But de Forest's true pa.s.sion lay in the cabinet of wonders he had a.s.sembled in his bedroom on Was.h.i.+ngton Boulevard: batteries, spark gap transmitters, electrodes-all the building blocks that would be a.s.sembled in the coming decade to invent the age of electronics.

For a budding innovator in wireless telegraphy at the turn of the century, the spark gap transmitter was the most essential of gadgets. Hertz and Marconi's original explorations of the electromagnetic spectrum had relied on spark gaps. The device employed two electrodes separated by a small gap. A battery attached to the electrodes supplied a surge of electricity, which caused a spark to jump from one electrode to the other, triggering a pulse of electromagnetic activity that could be detected and amplified by antennae miles away. Spark gap machines emitted a terse blast of monotone noise, perfect for sending Morse code.

On the night of September 10, 1900, de Forest was experimenting with his spark gap machine in the corner of his Was.h.i.+ngton Boulevard bedroom. Across the room, the red flame of a Welsbach burner flickered fifteen feet away. De Forest triggered a surge of voltage through the spark gap, and as the machine crackled, he could see the flame of the burner instantly change from red to white heat. De Forest later estimated that the flame's intensity had increased by several candlepower. Somehow, for reasons that de Forest could not explain, the electromagnetic pulse of the spark gap was intensifying the energy of a flame fifteen feet away. Watching that flame s.h.i.+ft from red to white planted the seed of an idea in de Forest's head: that a gas could be employed as a wireless detector, one that might be more sensitive than anything Marconi or Tesla had created to date.

De Forest had stumbled across a cla.s.sic slow hunch. In his autobiography, de Forest described the gas-flame detector as "a subject that had ever since been in the back of my mind." In the end, that hunch would mature into an invention that ultimately changed the landscape of the twentieth century, an invention that made radio, television, and the first digital computers possible. In 1903, he began a series of failed experiments with placing two electrodes in gas-filled gla.s.s bulbs. He continued tinkering with the model, until, several years later, he hit upon the idea of placing a third electrode in the bulb, attached to an antenna or external tuner. After a number of iterations, he used a piece of wire that had been bent back and forth several times as the middle electrode; de Forest called it the grid. Early tests showed that the device, which de Forest dubbed the Audion, proved far superior to other technology at amplifying audio signals without degrading the tuner's ability to separate out signals at different frequencies.

De Forest's creation would eventually be called a triode. Its three-electrode architecture would form the basis of the vacuum tubes that began to be ma.s.s-produced in the following decade. Radio receivers, telephone switchboards, television sets-all the communications revolutions of the first half of the century relied on some variation of de Forest's design to boost their signals. Initially employed for amplification, the vacuum tube turned out to have an unforeseen use as an electronic switch, enabling the highspeed logic gates of the first digital computers in the 1940s. When de Forest twisted the wire into the shape of a grid and placed it between those two electrodes, he was unwittingly opening up the adjacent possible for the a.n.a.lytical Engine that Charles Babbage had failed to produce sixty years before. The power of that new portal was apparent instantly: the first computer built with vacuum tubes, the mammoth ENIAC, ran calculations that helped develop the hydrogen bomb.

The invention of the Audion sounds like a cla.s.sic story of ingenuity and persistence: a maverick inventor holed up in his bedroom lab notices a striking pattern and tinkers with it for years as a slow hunch, until he hits upon a contraption that changes the world. But telling the story that way misses one crucial fact: that at almost every step of the way, de Forest was flat-out wrong about what he was inventing. The Audion was not so much an invention as it was the steady, persistent acc.u.mulation of error. The strange communication between the spark gap transmitter and the Wersbach gas burner flame turned out to have nothing to do with the electromagnetic spectrum. (The flame was responding to ordinary sound waves emitted by the spark gap transmitter.) But because de Forest had begun with this erroneous notion that the gas flame was detecting the radio signals, all his iterations of the Audion involved some low-pressure gas inside the device, which severely limited their reliability. It took another decade for researchers at General Electric and other firms to realize that the triode performed far more effectively in a true vacuum. (Hence the term "vacuum tube.") Even de Forest himself willingly admitted that he didn't understand the device he had invented. "I didn't know why it worked," he remarked. "It just did."

De Forest may have been the most erratic of the twentieth century's great inventors, but the error-p.r.o.ne history of his greatest success is hardly anomalous. The history of being spectacularly right has a shadow history lurking behind it: a much longer history of being spectacularly wrong, again and again. And not just wrong, but messy messy. A shockingly large number of transformative ideas in the annals of science can be attributed to contaminated laboratory environments. Alexander Fleming famously discovered the medical virtues of penicillin when the mold accidentally infiltrated a culture of Staphylococcus Staphylococcus he had left by an open window in his lab. In the 1830s, Louis Daguerre spent years trying to coax images out of iodized silver plates. One night, after another futile attempt, he stored the plates in a cabinet packed with chemicals; to his wonder the next morning, the fumes from a spilled jar of mercury produced a perfect image on the plate-and the daguerreotype, forerunner of modern photography, was born. he had left by an open window in his lab. In the 1830s, Louis Daguerre spent years trying to coax images out of iodized silver plates. One night, after another futile attempt, he stored the plates in a cabinet packed with chemicals; to his wonder the next morning, the fumes from a spilled jar of mercury produced a perfect image on the plate-and the daguerreotype, forerunner of modern photography, was born.

In the summer of 1951, a World War II Navy veteran named Wilson Greatbatch was working at an animal behavior farm affiliated with the psychology department at Cornell, where he was studying under the G.I. Bill. Greatbatch had long been a ham radio enthusiast; as a teenager, he had built his own shortwave radio by cobbling together the descendants of de Forest's Audion. His love of gadgets had drawn him to the Cornell farm because the psychology department needed someone to attach experimental instruments to the animals, measuring their brain waves, heartbeats, and blood pressure. One day, Greatbatch happened to sit at lunch with two visiting surgeons and got into a conversation about the dangers of irregular heartbeats. Something in their description of the ailment triggered an a.s.sociation in Greatbatch's mind. He imagined the heart as a radio that was failing to transmit or receive a signal properly. He knew the history of modern electronics had been all about regulating the electrical signals pa.s.sed between devices with ever more miraculous precision. Could you take all that knowledge and apply it to the human heart?

Greatbatch stored the idea in the back of his head for the next five years, where it lingered as a slow hunch. He moved to Buffalo, started teaching electrical engineering, and moonlighted at the Chronic Disease Inst.i.tute. A physician at the inst.i.tute recruited Greatbatch to help him engineer an oscillator that would record heartbeats using the new silicon transistors that were threatening to replace the vacuum tube. One day, while working on the device, Greatbatch happened to grab the wrong resistor. When he plugged it into the oscillator it began to pulse in a familiar rhythm. Thanks to Greatbatch's error, the device was simulating the beat of a human heart, not recording it. His mind flashed back to his conversation on the farm five years before. Here, at last, was the beginning of a device that could restore the faulty signal of an irregular heart, by shocking it back into sync at precise intervals. Within two years, Greatbatch and a Buffalo surgeon named William Chardack deployed the first implantable cardiac pacemaker on the heart of a dog. By 1960, the Greatbatch-Chardack pacemaker was pulsing steadily in the chests of ten human beings. Variations of Greatbatch's original design have now saved or prolonged millions of lives around the world.

Greatbatch's pacemaker is an instance where a great idea came-literally-from a novel combination of spare parts. Sometimes those novel combinations arrive courtesy of the random collisions of city streets or the dreaming brain. But sometimes they come from simple mistakes. You reach into the bag of resistors and pull out the wrong one, and four years later, you're saving someone's life. Yet error on its own is rarely enough. Greatbatch had his epiphany while hearing the reliable pulse of his oscillator because he'd been thinking about the irregular heartbeats as a signal transmission problem for five years. This, too, is a recurring pattern in the history of being wrong. The inventions of radiography, vulcanized rubber, and plastic all depended on generative mistakes that were generative precisely because they connected to slow hunches in the minds of their creators.

The British economist William Stanley Jevons, who had firsthand experience as an inventor himself, described the prominence of error in his Principles of Science Principles of Science, first published in 1874: It would be an error to suppose that the great discoverer seizes at once upon the truth, or has any unerring method of divining it. In all probability the errors of the great mind ex