It's wintertime in 2060. Traffic is winding its way through the mountain passes somewhere outside Tahoe. It's been a legal mandate for decades, now, that motor vehicles are strictly computer-operated on public roads.
Inside the cars, the occupants are sleeping, or chatting, or watching a movie, or generally doing anything but paying attention to the road -- which is normal. SkyNet's doing all the work. Since the self-driving mandate was finally enacted after years of constitutional challenges and not a few firebombings, traffic fatalities took a sharp plunge from almost 34,000 in 2009 to only a few thousand. Maybe a few hundred. (How intrinsically dangerous is driving?)
But today, suddenly, disaster strikes. On a two-lane road, a car in the inner westbound lane loses a tire and spins around to block the width of the roadway, directly in front of a column of cars approaching in the outer eastbound lane. Imagine that each car has identical and perfect knowledge of the positions and speeds of all of the other cars, the road conditions, and models for how each car will respond to control inputs. The network quickly realizes that the car at the head of the column cannot avoid colliding with the disabled vehicle without leaving the roadway, colliding with the guardrail and risking a plunge into a ravine below -- but the maneuver would avert any additional collisions. If the lead car stays on the roadway, it will be struck from behind by the second and then third cars in the column, leading to serious damage to each car, with risks to the occupants.
What should the algorithm controlling the lead car do?
Or imagine a different scenario, where the certain loss of any one of a set of four cars would avert a certain serious collision involving all four vehicles. Should one of the cars sacrifice itself? And how do you choose? Randomly? With a voting algorithm? Does it matter how many people are in each car?
What might some mitigating factors be? What if these incidents are so rare but so gripping that the autodriver companies start selling transferrable lifetime life insurance policies with each unit, that kick a cool $10 million dollars to your survivors if your car loses an election? Could there be a "suicide switch" that would let you opt to lose an election (in advance -- presumably there's no time for reaction in the moment)? Who would activate it?