Why feedback matters

It's 2017. A self-driven Tesla Model X is cruising down a freeway. The 'driver' looks up occasionally from the phone game he's playing. He has no reason to distrust the auto-drive function, and although the carmaker advises drivers to stay engaged, he ignores the beeps reminding him to. The car has been on auto-drive for 29 minutes, but he's been engaged for just 78 seconds.

In a flash, the car swerves and smashes into a concrete barrier, killing the driver.

March 28, 1979. Reactor 2 at the U.S. nuclear energy facility on Three Mile Island, Pennsylvania, overheats. Uranium escapes into the atmosphere and 80,000 people are evacuated. The situation is contained within days. Depending on whom you believe, the fallout may or may not have caused an outbreak of thyroid cancers over the next few decades. Back then, however, the meltdown was attributed to equipment malfunction and operator error.