When machines start making the calls
You open your inbox to find a message you do not remember writing. It is perfectly phrased, polite, efficient, and even includes your usual sign-off. The software that wrote it did not “think” in the human sense; it applied patterns learned from thousands of examples to predict what you might say next. That small act captures a much larger shift: decisions that once came from fixed algorithms are now being made by systems that learn, adapt, and generate on their own.
That was the thought that stayed with Saikumar Vellapareddy, a researcher in PwC’s Emerging Technology and R&D team. His curiosity began as a technical question about how machines learn to decide. As he traced the reach of automation— from early rule-based algorithms that simply follow instructions to modern AI models that interpret context and generate outcomes— he realised how easily people surrender control to systems they no longer fully understand.
In our research on outsourcing decision-making, we found that this quiet handoff is everywhere. Cars decide when to brake. AI hiring systems choose who moves forward. Financial platforms approve loans in seconds. Each of these is designed to help, yet collectively they mark a profound shift in agency. Decisions that once required human judgment are now made faster, more consistently, and more invisibly than ever before.
The autonomy curve
The path to autonomy follows a pattern. First, humans set clear boundaries and keep a close watch. Then systems begin to adapt on their own, responding to context and learning from data. Eventually, the oversight fades. The system becomes capable of acting independently, without waiting for permission or review. What began as support becomes control.
Our EmTech research team mapped this journey across three stages. In the first, called constrained autonomy, AI acts only within fixed rules. It is efficient but limited. The next stage, adaptive autonomy, adds learning and context, allowing systems to change course as conditions shift. The final stage, full autonomy, arrives when machines make and execute decisions entirely on their own. Each stage brings new value, but also less visibility into how and why decisions are made.
At first, this sequence feels like progress. The more we delegate, the faster and smoother our work becomes. But the gains in speed often come at a quiet cost. As machines learn to decide for us, we begin to lose not only control, but memory. Judgment starts to fade. Oversight becomes something we assume still exists, rather than something we actively maintain.
The illusion of oversight
The illusion of oversight is one of the most persistent challenges in autonomous systems. On paper, there are dashboards, audits, and compliance frameworks. In practice, the logic that drives machine decision-making is distributed across models, APIs, and vendors that few people can explain in full. The appearance of control replaces control itself.
Our research shows that organisations often mistake documentation for understanding. Policies are written. Reviews are scheduled. Yet the actual behaviour of an AI system is learned, not prescribed. When those systems evolve in production, their reasoning becomes harder to trace. A single change in data or model tuning can ripple through hundreds of dependent systems before anyone notices.
The more these decisions move beneath the surface, the more dangerous this illusion becomes. Accountability blurs. Responsibility spreads. In moments of failure, no one can fully answer the question of why a system decided the way it did. And when no one can explain the decision, trust in the outcome begins to erode.
Accountability as architecture
Vellapareddy believes the solution lies in how we define accountability itself. “We have to see accountability not as a policy, but as an architecture,” he said. “If a bad decision happens, we need to know who built the model, what data it used, and how its outputs were validated. That traceability has to be designed into the system, not added after something goes wrong.”
Our research supports that view and aligns with PwC’s Responsible AI framework: governance cannot be a static checklist, applied once a model is trained. It must be dynamic, built into the design from the start. True accountability depends on visibility at every layer— from who approved the use of data to who monitored model drift in production. When every decision is logged, traceable, and explainable, oversight becomes a feature of the system itself rather than a reaction to its failures.
In this framing, accountability is not a compliance burden but a foundation for trust. The organisations that build it into their design will move faster and with greater confidence because their systems can be explained. Trust does not come from accuracy alone. It comes from knowing who is responsible when the system gets it wrong.
What stays human
The real question is not whether we can build systems that decide for us, but what happens when we stop deciding altogether. Vellapareddy sees the danger not in the technology itself, but in how easily people adapt to its autonomy. “We are becoming lazy day by day,” he said. “Machines make our work easier, but if we stop using our own judgment, we lose a part of ourselves.”
In that reflection lies the central paradox of autonomy. The more we teach machines to think, the less we practice thinking for ourselves. And yet, the future of trust in technology depends on human intention— the choices we make about what to delegate, what to oversee, and what must always remain human.
As automation becomes the new architecture of decision-making, the most important decisions may not belong to the machines at all. They belong to us, in deciding how far we are willing to let them go.
Explore our services
Scale AI for your business