The enterprise AI conversation has spent two years focused on which model to choose. Google’s 2026 State of AI Infrastructure report, drawing on a survey of more than 1,400 senior IT leaders, makes a clear case that the more consequential question is whether your infrastructure can handle agents at all.
What Actually Happened
Google Cloud published its 2026 State of AI Infrastructure report in early July, surveying over 1,400 senior IT leaders globally. The headline finding: 83% of organisations said their infrastructure needs upgrades to support agentic AI workloads at scale. Only 17% have full confidence that their current tech stack can sustain mission-critical AI agents.
The cost picture is equally direct. According to the report, 62% of IT leaders are already seeing elevated inference costs driven by data egress, storage bloat, and idle specialised hardware in legacy systems. Nirav Mehta, VP of product management at Google Cloud, stated in the foreword that "Traditional architectures are cracking under this pressure." The report found that 96% of senior IT leaders now cite cost efficiency as the primary factor guiding AI infrastructure decisions — a shift from the capability-first framing that dominated AI procurement conversations in 2024 and 2025.
The shift from training to inference is part of the explanation. Inference now accounts for nearly half of all AI compute workloads compared with just 28% for training workloads. That ratio continues to move as enterprises shift from building models to running them in production.
Why Agents Break Legacy Architectures
The reason existing enterprise systems struggle with agents is not compute power — it is architecture. A standard generative AI deployment generates one inference call per user interaction: the user asks, the model responds. An AI agent completing the same request might trigger dozens or hundreds of downstream actions before returning a result: querying databases, calling external APIs, updating records, routing to other systems, and looping back through the model multiple times.
That pattern is fundamentally different from what enterprise infrastructure was designed to support. Legacy ERP systems, on-premises databases, and siloed data architectures built for human-paced workflows do not handle burst inference loads gracefully. The result is either degraded performance, runaway costs, or both — which explains why cost efficiency has moved to the top of the infrastructure decision criteria the Google report documents.
Agents consume significantly more tokens per completed task than single-turn generative AI, because the model must carry context across multiple reasoning steps. Enterprises that ran a satisfying generative AI pilot will find that the same infrastructure running agentic workloads at comparable volume produces materially higher bills, often without a corresponding increase in visible output.
The report concludes that hybrid multicloud architecture — a mix of on-premises and multicloud capacity — is emerging as the model best suited to absorb variable inference loads. Power consumption has also moved up the priority stack: 91% of senior IT leaders now factor energy consumption into hardware selection decisions, reflecting how far the cost conversation has shifted from capability to operational economics.
The Enterprise Lens
If your organisation is planning to deploy AI agents — to handle customer enquiries, automate order processing, manage document approvals, or assist with repetitive back-office work — the question to resolve before committing budget is not which platform to use. It is whether your current data and systems architecture can sustain the workload at the volume you are planning for.
Here is what that looks like in a real business context. An AI agent answering customer delivery questions needs to query your order management system, your logistics platform, and your CRM in real time, for every interaction, potentially dozens of times per hour. If those systems sit behind legacy middleware with slow response times or data egress charges, each agent interaction becomes both slow and expensive. The agent works in a pilot; the economics do not hold at scale.
The most useful conversation to have with your technology partner before scaling any agentic deployment is a straightforward infrastructure audit: where does your data sit, what are the integration points the agent will call, and what does the per-query cost look like at your target volume? A pilot at 50 interactions per day can look very different from a production deployment at 5,000. That gap is where most enterprise agentic AI projects stall — not because the model failed, but because nobody modelled the infrastructure cost before the rollout.
What to Watch
- Whether hyperscaler cloud providers introduce pricing tiers specifically designed for agentic workload patterns — flat-rate or agent-optimised compute pricing would materially change the cost calculus for enterprises currently on usage-based billing
- The production gap between organisations that upgrade infrastructure proactively versus those that attempt to retrofit after deployment — infrastructure readiness will increasingly determine which agentic deployments reach scale and which stall in the next 18 months
- How on-premises and hybrid infrastructure vendors respond to the Google report’s findings — the conclusion that 83% of enterprises need upgrades represents a significant commercial opportunity, and the architectural standards vendors propose over the next two quarters will shape enterprise deployment patterns for years
Sources