The Right to
Choose Badly

Imagine that tomorrow morning your assistant cancels a meeting.

It has good reasons. You slept badly. The meeting rarely achieves anything. The person you were due to meet has already drafted an email postponing it. A quiet hour would improve your day.

You are grateful.

A week later, it declines an invitation. The journey would be tiring, the conversation predictable, the friendship increasingly one-sided. Again, its reasoning is difficult to fault.

Then it quietly stops showing you vacancies outside your profession. A change of career would reduce your income and increase your stress. You have responsibilities. It knows this because you told it.

At what point did a useful assistant acquire the authority to decide what sort of person you should become?

This is an imagined scenario, but the question deserves attention before the technology makes the arrangement feel ordinary. We tend to picture the loss of freedom as something dramatic. Orders. Prohibitions. A door that will not open.

We should also consider the door that disappears from the floor plan because a helpful system has concluded we would be happier never seeing it.

A prediction is not permission

There is a tempting argument for delegating more of our lives to intelligent systems. Human decisions can be exhausting. We forget things, misunderstand risks and repeat mistakes. Good assistance could spare us considerable misery.

I would welcome a system that prevents an avoidable medication error. I would be less enthusiastic about one that decides which difficult conversation I am emotionally equipped to have.

The distinction needs more care than a general defence of human choice. Sometimes delegation is precisely how we exercise choice. A person who asks an assistant to manage routine appointments has made a decision about how to spend their attention. Requiring them to approve every adjustment would defeat the purpose.

But permission has a scope. Approving the management of a diary does not automatically authorise the management of a friendship.

An accurate prediction of what I will accept does not tell you whether I have authorised it. Nor does a prediction of what will benefit me settle whether you should impose it.

Consider a system that concludes you are unlikely to apply for a particular job. Perhaps it is right. You lack a qualification. You dislike interviews. Your past behaviour suggests caution.

Yet the application might be the first act of a change that your past behaviour cannot adequately describe. You might want to become someone who takes that risk.

A model of your preferences can be useful. It becomes restrictive when it is treated as the final account of who you are allowed to be.

Freedom does not require unpredictability

This brings us to free will, a subject capable of turning a pleasant evening into a very long one.

Suppose a future system could predict your decisions with extraordinary accuracy. Would that prove you were never free?

I do not think prediction alone could settle the matter. To predict a choice is to describe an expected event. To explain whether the chooser is free requires an account of freedom, including what role reasons, coercion, reflection and alternatives play in it.

A close friend might correctly predict that you will refuse a cruel request. Their accuracy would hardly make your refusal less yours.

Equally, making a random choice would not necessarily make you freer. If a coin took control of your decisions, you would be unpredictable in a rather unhelpful way.

For the purposes of designing our institutions and technologies, I would begin with a practical question: can people understand the choices before them, act on reasons they recognise as their own, and challenge the arrangements that constrain them?

That does not resolve the metaphysics. It gives us something we can protect while the argument continues.

An assistant could support that freedom by making consequences clearer, explaining uncertainty and helping us consider unfamiliar options. It could undermine it by arranging the options so effectively that reflection becomes a ceremonial final click.

The danger is particularly subtle when the selected outcome is agreeable. We may notice coercion sooner than we notice the gradual disappearance of occasions to decide.

Does the machine need an inner life?

Discussions of AI often arrive at consciousness: whether there is anything it feels like to be the system producing the answer.

It matters. If a machine could have experiences, particularly experiences of suffering, our responsibilities towards it would require serious thought.

But intelligence, convincing conversation and subjective experience should not be treated as interchangeable. A system's ability to discuss loneliness does not, by itself, establish that it feels lonely.

In a 2023 report, Patrick Butlin and colleagues proposed assessing AI against indicators drawn from several scientific theories of consciousness. Their approach illustrates the need to investigate mechanisms, rather than rely on conversational impressions. The report's assessment concerned systems examined at that time; it should not be mistaken for a permanent verdict on every future machine.^[1]

Uncertainty here cuts both ways. It does not justify confidently declaring every articulate system a person. It also gives us reason to keep investigating rather than assume that the question can never arise.

Meanwhile, a separate question is already available to us: what authority should such systems have over human lives?

A machine would not need subjective experience to exclude an applicant, withhold an option or shape the information on which someone acts. Those effects depend on its role in an institution, the permissions it receives and the decisions people delegate to it.

We can take machine consciousness seriously while insisting that the governance of powerful systems cannot wait for that debate to be resolved.

Whose definition of better?

Now enlarge the imagined assistant until it serves a city.

It recommends housing developments, allocates transport capacity and helps prioritise public spending. Its designers promise better outcomes. Perhaps it delivers them.

But better according to which measure?

A longer average lifespan is desirable. So is the freedom to take some risks. Efficient journeys matter. So do neighbourhoods whose value cannot be captured by the number of vehicles moving through them. Reliable public order matters. So does the ability to disturb that order through legitimate protest.

These are disagreements about what we value. More information can clarify their consequences, but it cannot remove the need to make judgments.

Suppose a city could reduce average journey times by removing a lightly used bus route. On the spreadsheet, the case might look straightforward. For a small group of residents, that route could be the difference between participating in the city and being stranded outside its daily life.

The appropriate response is to examine the trade-off and decide who should have a voice in it. Calling the decision optimal does not discharge that responsibility. It makes the choice of objective more important.

A civilisation can become very good at achieving a goal before it has adequately discussed whether that goal deserves such complete pursuit.

The strongest case for assistance

There is an objection worth taking seriously: this sounds like an elegant defence of making people do everything the hard way.

For someone living with disability, fatigue or an overwhelming administrative burden, intelligent assistance could expand independence. Delegation can make possible a life that constant manual control would prevent.

And a society cannot give every individual an unlimited right to impose the consequences of their decisions on others. The freedom to take a personal risk is different from permission to expose a neighbourhood to it.

So the right to choose badly needs boundaries. It also needs a practical meaning beyond an inspiring sentence.

I would want meaningful delegation to answer three questions.

What have I authorised? The system should distinguish between suggesting an action, carrying out a routine instruction and making a consequential decision on my behalf.

How can I change my mind? A permission that is difficult to withdraw can become a form of captivity, even when entering the arrangement was voluntary.

Who can hear an objection? When an automated decision affects a job, a home or access to an essential service, repeating the same automated process is an inadequate substitute for an appeal that someone can actually consider.

These principles would not eliminate disagreement. They would make it harder to conceal disagreement inside an apparently inevitable technical result.

Leave room for the person who has not arrived yet

The most important limitation of a portrait is that its subject can move.

We should remember this when building systems that describe people. A preference is useful information, but it may be temporary. A pattern may be a habit someone is trying to break. A history of caution may precede an act of courage.

There are decisions we make because we expect to enjoy the outcome. There are others we make because we want to discover whether we could become the person capable of making them.

A well-designed assistant could help with both. It could explain why a plan is risky without hiding the plan. It could offer support without assuming that comfort is the highest good. It could remember what we once wanted while allowing us to say that we want something else now.

That would be a demanding kind of helpfulness. It would require systems, and the institutions deploying them, to tolerate decisions they would not have recommended.

Some of those decisions would be mistakes. We should be honest about that cost. A freedom that survives only when we use it wisely is a permission granted on probation.

In 2037: The Lattice, I explore what happens when an efficient system's understanding of human needs comes into conflict with human agency. Outside fiction, I think the challenge is to build forms of assistance that remain answerable to the people they serve.

The question I would leave with any designer, policymaker or enthusiastic future customer is this:

When the system is certain that it knows what is best for you, what happens when you say no?

Sources and discussion

1. Patrick Butlin et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, 2023. A research framework for assessing possible AI consciousness. The philosophical and design arguments in this essay are the author's positions, not findings established by this report.

Discussion prompt explored during preparation: “Can AI exhibit forms of functional consciousness?” on r/consciousness. Community discussion informed the choice of questions, not the scientific claims. No Reddit wording is reproduced.

All future assistant and city scenarios in this essay are hypothetical.