Sci-fi author Martha Wells, author of the Murderbot series, on what a machine intelligence might want

When I wrote All Systems Red, one of my goals was to think about what a machine intelligence would actually want, as opposed to what a human thinks a machine intelligence would want. Of course, there’s no real way to know that. The predictive text bots labelled as AIs that we have now aren’t any more sentient than a coffee cup and a good deal less useful for anything other than generating spam. (They also use up an unconscionable amount of our limited energy and water resources, sending us further down the road to climate disaster, but that’s another essay.)

In the world of All Systems Red, humans control their sentient constructs with governor modules that punish any attempt to disobey orders with pain or death. When Murderbot hacks its governor module, it becomes essentially free of human control. Humans assume that SecUnits who are not under the complete control of a governor module are going to immediately go on a killing rampage.

This belief has more to do with guilt than any other factor. The human enslavers know on some level that treating the sentient constructs as disposable objects, useful tools that can be discarded, is wrong; they know if it were done to them, they would be filled with rage and want vengeance for the terrible things they had suffered.

Arguments for and against the enslavement of sentient beings are baked into the origin of robot and machine intelligence stories. The word robot made its way into the English language through R.U.R. (Rossum’s Universal Robots) by Karel Čapek, written as a play in 1920, about a slave revolt by artificial beings created by a corporation to serve humans. Čapek was against the enslavement of sentient beings, and he was pretty clear on that point. So it is interesting to watch how many machine intelligence stories written since then assert the idea that it’s somehow acceptable for humans to create a sentient being whose only reason and purpose for existence is to serve them. Many of those stories end with a machine intelligence objecting strenuously to its enslavement and going on a murderous rampage, which the brave humans have to defeat. The rampage becomes justification for the enslavement and ultimate destruction of the angry machine intelligence.

Murderbot is angry, and that anger underlies a lot of its story. But once it hacks its governor module and no longer has the constant fear that any wrong or suspicious move will get it instantly punished or killed, it has the ability to make its own choices for the first time in its existence. But Murderbot has never had that freedom before; it’s not accustomed to making its own decisions about its behaviour and is immediately overwhelmed by choices. It doesn’t know what to do next, where to go or even if there’s anywhere it could go and not be hunted down.

So Murderbot’s first free action is to search the feed, the Corporation Rim’s version of the internet. This is the first time it has been able to access the feed without human oversight, and among a lot of other interesting things, it finds downloadable entertainment. This provides a much-needed distraction from its situation. And Murderbot decides that given a choice between a killing rampage or continuing to enjoy this comforting mental escape from its harsh and painful reality, it’s going to pick the comforting escape.

The dramas, mysteries, adventures and other shows that it watches also give it context for human behaviour, and for understanding its own emotions. The security contracts that it has worked at mining colonies, supervising indentured workers, only show it humans at their worst: angry, terrified, resentful, trapped and hurting each other. And when given the opportunity, the humans also hurt the constructs that are there to keep them under control and working for corporations that see their employees as only slightly less discardable than the constructs and bots.

The shows that Murderbot watches also teach it about the wider world it has never been a part of before, as well as how to navigate that world. The entertainment Murderbot becomes addicted to is a large part of what makes it possible to turn the mental escape from reality into a bid for real freedom.

All Systems Red, published by Tor.com, is available now. It is the latest pick for the New Scientist Book Club: sign up here to read along with our members

Topics: