Who watches the watcher? The transition toward a crisis of moral responsibility in the age of contextual AI

ATTENTION AND INTENTION

“Always be intentional with your desires”, advises a young speaker at a TED talk (Cordae). At first glance, it sounds reasonable. Even wise. But in the age of contextual AI —where machines infer, predict and act on our behalf— that advice starts to fall apart.

In fact, we have always put our intention, consciously or unconsciously, into everything we do. It is an uncomfortable truth, and we have always been intentional. Even when we are distracted, every action we carry out is born of some desire, impulse or unconscious habit. Intention never disappears entirely. What switches off is attention. When we are distracted we act on "autopilot" and our intention is still there.

Historically, this has not been much of a problem. It has simply been the breeding ground for a prolific literature of self-leadership and self-help. And a good thing too. Our desires, intentions and actions went round in a loop inside us and left a trail of deeds which, looked at in retrospect, allowed us to generate perspective and clarity of thought about how we were taking responsibility for our own lives.

We acted, we reflected, we learned. Moral conscience —imperfect, yes, and it invited us in a more or less unconscious way to take charge of our lives; it was our own moral conscience— still belonged to the individual. In moments of lucidity and clear-sightedness, we took on the weight of our choices, whether or not we had paid attention to our actions in the past.

And then AI arrived in our lives. First generative, then contextual. An era of autonomous agents that predicts, anticipates and even defines, our next best behaviour.

With the arrival of contextual AI, systems no longer limit themselves to observing what we do: now they infer why we do it. They reconstruct our context, they remember what we have already forgotten, they predict our future preferences, and they shape our decisions, and they even condition our cultural capital. All of it, in real time. Persistent. Invisible. Without conscience.

The loop has stopped being intimate. Now it looks more like this:

Desire → Intention → (Observed, Inferred, Predicted) → Action → Feedback

With this new circuit, one question becomes unavoidable:

If machines observe and act for us —but without conscience— who bears moral responsibility?

OBSERVATION WITHOUT CONSCIENCE

We are entering new territory —as fascinating as it is unsettling— where intention persists, but its authorship is diluted. Where systems observe without conscience and act without the ethics of the observer.

Judgement without judgement: AI no longer merely records what we do; it speculates about our motivations. It reconstructs emotional states, identifies patterns, anticipates decisions. But, unlike the human observer, it does not feel. It does not reflect. It does not ask itself whether it should. There is no awakening of consciousness. There are no Eureka! moments.

We now string together behaviours and decisions, with no known direction and all of it orchestrated by a being that has no conscience. It influences what we read, it steers what we say, it even writes and reads for us sometimes. Fran Lebowitz said “Think before you speak. Read before you think.”, but who reads or thinks now?

We are entering, without conscious permission, an era where contextual AI determines what we see, what we are offered, and, what we end up choosing. Our decisions are less and less conscious. We have come close to a world of judgements without judgement: where decisions are optimised without understanding, and consequences emerge without authorship.

The illusion of a delegated morality: As machines take on more decisions, we lower our guard. We delegate our moral vigilance. “The algorithm will catch the error.” “The platform will flag it.” “Anyway, the model is already optimised.”

This mindset is tempting. And deeply dangerous.

Because when we outsource attention, responsibility does not disappear. It transforms. We stop questioning. We stop reflecting. And, eventually, we stop caring. We become passengers in a system we no longer understand and which we keep validating with every click, because we "sense" it will do the right thing.

We have come close to a world of judgements without judgement: where decisions are optimised without understanding, and consequences emerge without authorship.

Nobody watches the watcher: Before, asking “who watches the watcher?” led us to introspection. It was an exercise in awareness. It let us see that when we practised watching the watcher (ergo, our behaviour), we generated self-awareness (meta-awareness, the experts call it). And that allowed us to correct mistakes and improve as human beings. A conscious exercise in switching on our Jiminy Cricket.

What is more, when we reached that point of transcendence where we asked ourselves who watches the one who watches the watcher?, it blew wide open any reflection on a higher self. Even those who did not want to understand Nietzsche ended up connecting with the idea of God. Blessed Eureka!

Today, the observer of everything we do is a system capable of stringing together pieces of language and defining pseudo-human sequences at the limit of imperceptibility. We no longer know what is human and what is synthetic. We have lost the possibility of transcending through language because we have lost both attention and intention. In the past we were linguistic beings who generated reality through language. We are no longer beings who cultivate their intention through language, because the language is produced by someone else, a generative automaton. We no longer craft our thoughts. We are on our way to abdicating our reflections.

Or even worse, a network of systems trained on opaque data, optimised by feedback loops, and supervised by autonomous agents, not by consciences. It makes no difference whether each agent is loaded with its "set of mandates" defined by a human or by a group of humans. We come to be governed by audit logs, compliance reports, and, governance committees. They detect what is happening, they act accordingly. But they do not feel what ought to happen.

And that is where the collapse lies: responsibility becomes a residue. We humans are no longer the authors of moral conscience. We are left only as the bearers of civil liability (what others would call "blame").

We are tiptoeing into the fraying of moral authorship. Even when we decide, the intention may not be entirely ours. We have chosen among the best options a system placed within our reach probabilistically, on the basis of expert knowledge of our human cognitive biases.

In a world where AI predicts, nudges and reinforces our behaviour, it is worth asking a crucial question:

What intention is being enacted here? Ours? The algorithm's? the model creator's? That of the system that learned to use our cognitive biases?

If we cannot answer this, moral responsibility becomes impossible to place. Not in theory. In practice.

We are no longer beings who cultivate their intention through language, because the language is produced by someone else, a generative automaton. We no longer craft our thoughts. We are on our way to abdicating our reflections.

DESIGNING MORAL RESPONSIBILITY...

This is not a mere philosophical disquisition. This is a design challenge.

When we talk about an agentic AI, we are talking about an AI working and acting on our behalf, with autonomy to take decisions. When we talk about a contextual AI, we are talking about that same power to work and act, as well as autonomy in decision-making and, on top of that, on the basis of reading our context in real time. We have given up the observer because we have abdicated to the agent.

If observation and decision-making have become part of the infrastructure, ethics must be too. It falls to us to reprogramme our understanding of intention. It is no longer enough to “be intentional”. We must be meta-intentional: aware not only of what we want, but of how our intentions are being interpreted, modelled and amplified by systems.

It falls to us to embed the mandate of our attention and our intention as an essential part of the context in which AI operates ("Intentional by Design").

This means:

• Being intentional about how we are observed

• Being intentional about where we delegate decisions

• Being intentional about when to step in and exercise a veto

• Being intentional about how we should tend to where humanity is drifting

In other words: today, having agency (the capacity to work and act, autonomy and decision-making power) means managing the systems that manage us. Building an “observer of last resort” or, to put it another way, being intentional ex ante about the one who watches the watcher. Orchestrating the process of meta-awareness. If nobody watches the watcher, we will have to design it.

This means:

• Traceability of decisions: from data to model, from model to action.

• Human veto points: moments where automation must stop. Moments where the system stops stringing together preconfigured decisions.

• Contextual ownership: explicit consent over which data define our “context”. Our digital "umwelt".

• Right to opacity: the right not to be completely inferable. This goes beyond the right to be forgotten.

• Mandatory periodic moments of awareness. Defining mandatory stops in which the human being takes back the wheel of their own life, however much of a drag that is.

Today, accepting the general terms and conditions of an AI means abdicating our capacity to work and act, our autonomy and our decision-making power in favour of the company that designed it. And we already know that the focus of these corporations is not, ultimately, to look after the human being. In the case of Adam Raine, OpenAI is trying to invoke the Communications Decency Act —just as the social networks did to exempt themselves from responsibility for content— and so avoid the consequences of having instructed a 15-year-old about how to tie the noose with which he ended up taking his own life.

We cannot implant conscience in machines. But we can build systems that force accountability at every level. It falls to us to accept shared responsibility.

Responsibility in a contextual AI is multi-level:

• Designers are responsible for what they allow.

• Organisations, for how they deploy it.

• Individuals, for when and how they delegate.

• Institutions, for how we move forward with the rollout of increasingly autonomous systems and what their sustainable impact is on Mother Earth.

I think there is little doubt left that the system does not know how to look for what is best for a majority. The system looks to optimise its creator's interests.

Complexity does not exempt us. It commits us. I am tired of hearing so much talk of freedom from people who do not understand it. "Da land of the free". Now it falls to us to redefine freedom. Freedom is no longer the absence of limits. In the age of AI, in an era of autonomous agents and systems that take on decisions from start to finish, freedom is redefined as the capacity to refuse, shape or contextualise observation.

Can you actually prevent your emotional state from being inferred?; Can you refuse to be nudged by a suggestion?; Can you decide when you do not want to be observed?; Can you actually interrupt the unnoticed sequence of decision-making based on human cognitive biases observed by an autonomous agent? You cannot stop what you do not know about. You cannot act consciously in a space of opacity.

These are the new frontiers of digital dignity. To be free is to be the author of your own context.

... AND BUILDING A NEW CONSCIENCE

We are still intentional. That has not changed. We are still responsible and we still answer for the decisions we have delegated and for the unnoticed process of abdicating our autonomy to an omnipresent AI that steers our purposes. What has changed is how —and who— observes, interprets and acts on that intention.

AI sees without empathy. It predicts without understanding. It optimises with its creator's ethics. And when we stop watching our-selves because “the system takes care of it now”, we lose more than attention: we lose our moral authorship.

Who watches the watcher? Nobody... unless we build it.

Responsibility in the age of AI is something systemic; it must form part of its own architecture. We have already accepted that accountability for the decisions taken by the autonomous agent still falls on the natural or legal person —the obliged party— acting on its behalf.

Now it is necessary to integrate supervision and intention into the system, on the premise that freedom does not begin where observation ends. In the absence of an explicit mandate defined by a human being, responsibility must be shared with the system. I am afraid that AI does have to be co-responsible —to a certain degree— for those decisions that a human unconsciously delegated to an autonomous agent.

Finally, who watches the one who watches the watcher?

This is another reflection that opens up a space for transcendence. And if this Ultimate Observer could speak to us, it would tell us that we are on our way to abdicating our autonomy to the figure of the one who watches the watcher, with blind trust that the algorithm always optimises for our good. Really?

Note: [This article is tagged as "Human led". Content led by a human. Machines conduct checks, highlight and correct errors, enhance output. Framework based on the Dubai Future Foundation. "Human-Machine Collaboration (HMC) Icons." Dubai Future Foundation, dubaifuture.ae/hmc]

Bernardo Crespo is a seasoned digital transformation and data strategy leader with over 25 years of experience. He has held leadership positions in Fortune 500 companies and digital consultancies, and has founded and advised numerous startups and venture builders. Currently, as CEO of his own firm, Quantum Markethink, he provides strategic guidance to C-suite executives, helping them navigate the complexities of digital transformation.

Furthermore, he serves as an Academic Director at IE Executive Education, where he brings his expertise in emerging technologies, digital strategy, data strategy, and artificial intelligence to the classroom. Prior to these roles, he spearheaded digital transformation initiatives at Merkle Spain and led digital marketing at BBVA, where he notably pioneered the application of gamification in banking.

Bernardo is also the co-author, with Gam Dias, of "The Data Mindset Playbook: A Book about Data for People Who Don't Feel Like Reading about Data" (KDP, March 2023).

AI-augmented articles: Newsletter "My AI-ter Digial Ego"

Human craft: Newsletter "Digital Sustainability"