Defining the CPU benchmark for agentic AI
The rise of Agentic AI—systems that can autonomously handle complex, multi-step tasks—has ignited a firestorm in the world of computing. Yet, despite the phenomenal capability of these AI agents, understanding how they truly stress the underlying hardware remains a challenge, lacking solid, repeatable benchmarks. To truly unlock the performance potential of these intelligent workflows, researchers are now diving behind the curtain, isolating the performance contributions of the CPU and memory within the agentic loop.
To tackle this complexity, the team developed an internal testing methodology designed to evaluate how these components contribute to the agentic process. The goal is to move beyond simple model performance and understand the true efficiency of the system when an agent is executing a sequence of tasks.
What exactly is an agentic loop? Think of an AI agent as a little worker, equipped with a Large Language Model (LLM), that can execute tasks by utilizing tools. These tools might involve reading files, querying a database, or running code. When an agent needs to solve a problem, it often engages in a cycle: it identifies a need, calls a tool, receives the output, re-evaluates the situation, and calls another tool. This iterative process—the agentic loop—is what defines advanced AI work.
For example, if an agent is tasked with debugging a Python script, it might first call a “list files” tool, then “read file,” then “run command,” and if the file isn’t fixed, it loops back to the beginning. This dynamic interaction with tools forms the core of agentic intelligence.
When measuring this behavior, researchers focused on how performance scales across the CPU as agent sessions increase. The testing methodology was engineered to provide an ideal throughput, allowing each chip to maximize agent sessions while remaining interactive. This approach allows observers to see how CPU and memory performance diverge when running complex, multi-turn agentic flows.
To test this, the team utilized specialized hardware, including an Nvidia DGX Spark (GB10) featuring a powerful Blackwell GPU and a Grace CPU. Crucially, the testing sought to understand resource contention. While the GPU often dominates the time spent on model inference, the CPU is still critical for executing the tool calls, code compilation, and overall task management. The research found that even demanding CPU tasks often represent only a small fraction of the total computational time when a powerful GPU is involved.
Instead of focusing on sequential execution, the team introduced the concept of concurrency. By running multiple agent sessions simultaneously on the same system, they could observe how resource contention manifests. This test design allowed them to see how agentic work stresses the CPU in a more realistic, concurrent environment, uncovering subtle bottlenecks that sequential testing missed.
The benchmark involves running custom agents, each with a unique set of tasks defined by specific personas—such as an Analyst who queries a database or a Coder who fixes bugs. These personas demonstrate complex workflows, ranging from document extraction and web browsing to coding and research.
The results focus on measuring sessions per second, which determines the overall throughput of agentic work on the CPU. The findings highlight the importance of finding an “ideal throughput” where task completion times stabilize, revealing the point where increased session counts begin to degrade performance. This provides critical insight into the hardware characteristics needed to support the next generation of autonomous AI agents.
As agentic AI continues to evolve, this research provides a vital starting point. By understanding the intricate relationship between CPU, memory, and accelerator performance, developers can make much more informed decisions about where future AI hardware and software development should focus their efforts.
Buy on Amazon
Credit
