What if the cost of machines that think is people who don’t?

The Glass Cage is an important counterpoint to the dominant automation-at-all-costs mindset of Silicon Valley. That more automation is better is not as obvious a conclusion as many of us would like to believe. Carr is definitely not anti-technology though. This book is level-headed in discussing the positive and negative trends in automation, backed by a large amount of research. From pilots to doctors to inuit hunters, Carr present a comprehensive overview of automation across society today.

Three themes in particular stuck in my mind.

Sharp Tools, Dumb Minds

Carr summarises a number of studies that have measured the effect of computer aided learning.

One 2004 study presented two groups of people with the Missionaries and cannibals problem. The first group used a computer program that offered step-by-step guidance and prompts for valid moves. The second group used a basic computer program that offered no such assistance. The aidless group was slower to get started, but excelled in the later more complicated stages of the game. The first group, “by contrast, often became confused and would ‘aimlessly click around.’” This same lack of fundamental understanding has been documented in the use of real world expert systems, in professions from accounting to medicine. This effect is compared later in the chapter to a calculator:

If you use the calculator to bypass learning, to carry out procedures that you haven’t learned and don’t understand, the tool will not open up new horizons. It won’t help you gain new mathematical knowledge and skills. It will simply be a black box, a mysterious number-producing mechanism. It will be a barrier to higher thought rather than a spur to it.

Expert systems do help experts be more effective, but you need to be an expert first.

Straightjackets Of The Mind

Several sections are devoted to medical systems and GPS navigation, two of the more common automation aids in widespread use today.

In the medical field, automated diagnosis has led to increased testing, bloated and unhelpful medical records, and excess billing for procedures that would historically have been routine.

One of the common assumptions about electronic records is that by providing easy and immediate access to past test results, they would reduce the frequency of diagnostic testing. But this study indicates that, as its authors put it, “the reverse may be true.” By making it so easy to receive and review test results, the automated systems appear to “provide subtle encouragement to physicians to order more imaging studies,” the researchers argue.

Further, when using common electronic systems “doctors can begin to display ‘screen-driven’ information-gathering behaviors, scrolling and asking questions as they appear on the computer rather than following the patient’s narrative thread.” As suggested by the earlier studies on learning, this effect is particularly detrimental in stunting the growth of inexperienced doctors.

Diagnostic accuracy is only increased when expert systems provide suggestions to the doctor for things they may have overlooked, rather than recommending an actual diagnosis. But these must be set at a threshold low enough that the alerts are actually read. The current trend is to over-suggest: “physicians routinely dismiss about nine out of ten of the alerts they receive.”

Simarly, GPS systems are stunting the ability of people to navigate. In one study,

Some of the subjects were given hand-held GPS devices; others used paper maps. The ones with the maps took more direct routes, had to pause less often, and formed clearer memories of where they’d been than did the ones with the gadgets.

Carr however is more concerned by the alienation this causes rather than the direct lack of skill.

The automation of wayfinding serves to ‘inhibit the process of experiencing the physical world by navigation through it.’ […] But while we may no longer have much of a cultural stake in the conservation of our navigational prowess, we still have a personal stake in it. We are, after all, creatures of the earth. […] It provides a sense of personal accomplishment and autonomy, and it also provides a sense of belonging, a feeling of being at home in a place rather than passing through it. […] the more you think about it, the more you realize that to never confront the possibility of getting lost is to live in a state of perpetual dislocation. If you never have to worry about not knowing where you are, then you never have to know where you are.

For someone who prides himself on hand-drawn maps to find his destinations, this bit of ego-stroking was not unwelcome.

Utopian Promise

The concluding chapter was also perhaps the scariest. Carr considers possible futures, particularly ones in which the relentless drive to automation frees society for the pursuit of leisure. He does not find this scenario plausible:

It strains credulity to imagine today’s technology moguls, with their libertarian leanings and impatience with government, agreeing to the kind of vast wealth-redistribution scheme that would be necessary to fund the self-actualizing leisure-time pursuits of the jobless multitudes.

Quoting Hannah Arendt’s The Human Condition, automation confronts us with “the prospect of a society of laborers without labor, that is, without the only activity left to them. Surely, nothing could be worse.”

I recommend The Glass Cage highly for anyone working the technology sector. It covers many more issues than the three I covered here, including driverless cars, historical trends, ethics (particularly in regard to military automation), de-humanization, aeronautical automation, and human-centered automation. We wield a great amount of power with the tools we build, and we have a responsibility to weild it wisely.