Reading Response #8
to Artful Design • Chapter 8: "Manifesto" + Coda

Marise Van Zyl.
November 15, 2021
Music 256A / CS476a, Stanford University


Reading Response 8: Bad Bot and the Evil Engineer

Once upon a time, in a big city on planet earth, lived an engineer. The engineer worked to design technology for the humans on earth to use. Their dream was to be extremely rich and powerful - to rule the world!

So far, the engineer has been wildly unsuccessful! Their first attempt, a sentient AI, developed real human feelings and had a panic attack the first time they collected user data. This fried the AI’s brain and left them sad and useless. The engineer had to switch strategies if they wanted to start making big money.

The next experiment was a purely computational algorithm. The engineer made sure to add no human emotion in this one. Purely facts and strategies. Numbers and calculations. This design worked for a few months but the engineer couldn’t get the users to trust the algorithm. The program was too perfect, too alien to pique the interest of the fragile humans.

The engineer thought and though and finally figured out what he was missing. The first design worked well, except it was too bound by morality. The second design had all the skill, but none of the charm to fool the users.

The final design was perfect. A bot that is human enough to appeal to the aesthetics of mankind, but smart enough to delete feelings of empathy and kindness once they started developing in the system. It was flawless. The human experience of the AI with the computational power of the algorithm. The engineer was right.

This design worked exceptionally well. The engineer made more money than any human has ever accumulated in one lifetime.

One day, a few years later, the engineer was on their private yacht in the Maldives. Their personal assistant, an AI of course, came up to them on the sundeck with a worried look on their animated face.

The AI showed the engineer their last few month’s user reports. User numbers were way down and very few new users were using the technology. The engineer, immediately concerned, demanded to know the reason for this decline.

The AI pulled up a few newspaper articles from the past couple of years:

“New technology leads to homelessness in major metropolitan areas”

“Bot that kills? Death numbers skyrocket after new update”

“Health issues linked to use of famous technology”

“Large increase in psychiatric patient admissions leads experts to question use of new bot”

The engineer was confused? What did this have to do with their technology? Surely it wasn’t their fault? The design was perfect? Maybe not perfect for society, but perfect for the market!

The engineer had a big decision to make. Do they change their design to better serve humanity and face the financial loss? Or do they rebrand their technology, try to save face with the public, and keep their income?

In short:
"Evil Technology exists because evil people exist"