NIST Proposes First Standardized Humanoid Robot Benchmark Since 2015

The Problem: No Way to Verify What Humanoid Robots Can Actually Do

Ten years have passed since the DARPA Robotics Challenge, and the humanoid robotics space looks nothing like it did then. Tesla's Optimus, Figure, Agility, Apptronik, and Unitree have collectively attracted billions in investment. Yet, as Aaron Prather, director of the Robotics & Autonomous Systems Program at ASTM International, pointed out on LinkedIn: "There is still no agreed-upon way to measure what any of them can actually do. Marketing videos have filled the gap."

This is a problem for anyone making purchasing decisions, regulatory frameworks, or research investments. Without standardized benchmarks, comparing claims across platforms is essentially impossible. The industry has been flying blind.

The Solution: A Low-Footprint Performance Baseline

NIST's Intelligent Systems Division released a proposed baseline performance benchmark last month. The framework describes "a low-footprint set of locomotion and manipulation tasks" using previously defined, standardized test methods and performance metrics. The institute built this proposal on its prior collaboration with DARPA evaluating humanoid capabilities across industry and academia.

The benchmark targets four core capability areas:

- Domain-agnostic, basic humanoid robot mobility and manipulation/dexterity capabilities - Coordinated capabilities combining locomotion and manipulation tasks - Whole-body awareness and control through confined-space manipulation tasks - Minimal reasoning, task and scene understanding, and decision making

NIST designed the baseline benchmark apparatus in collaboration with industry and the research community. The institute plans to build a limited number of testing apparatuses for free distribution to U.S. humanoid robot manufacturers and established regional testing facilities. NIST will also publish the designs and 3D models of the apparatus for use as a physical or virtual testbed for robot training and control development.

Robot manufacturers can receive an apparatus (pending availability) to run their own tests, or they can test their robots at NIST or a participating facility. Test results will be collected under pre-approved data-sharing agreements to protect intellectual property.

This approach mirrors Fraunhofer IPA's own benchmark proposal, released earlier this month, which outlined six criteria for humanoid safety and development. The fact that multiple organizations are independently converging on similar frameworks suggests the market recognizes this gap.

What This Means for Manufacturers and Researchers

NIST is actively seeking input on several key questions before finalizing the benchmark:

- Do these tasks sufficiently exercise minimum humanoid capabilities? - Are there tasks that would better exercise whole-body-control and/or loco-manipulation? - What additional constraints would manufacturers need for testing? - Are companies willing to help design the benchmark and participate in testing? - Is there interest in becoming a participating test facility?

The institute plans to aggregate results to showcase the state of the art of humanoid robot capabilities. This aggregation could finally provide the cross-platform comparison data the industry desperately needs.

Interested parties can contact Dr. Benjamin Beiter or Dr. Kamel Saidi at NIST for more information or to participate in the development process.

My Take

This matters because we're approaching the point where humanoid robots will move from pilot deployments to volume purchases. Without standardized performance data, procurement teams are relying on vendor-provided demonstrations that may not reflect real-world performance. NIST's involvement adds federal credibility and access to established measurement infrastructure. The open design approach means smaller manufacturers aren't locked out of the benchmarking process. Whether this framework becomes the actual standard depends on industry participation. If the major players commit, we could have actionable comparison data within 18-24 months.