METR and Redwood Find Monitoring Blind Spots in OpenAI's Safety Protocols

Up to 25 percent of internal agentic traffic from OpenAI’s GPT-5.5 class models and above may have operated without any monitoring coverage during a recent three-week audit. This is the central finding of a report released by METR and Redwood Research, two independent organizations that spent July 21 through August 11 on-site at OpenAI. The researchers had full access to the company’s codebase and internal documentation, and they interviewed approximately a dozen employees to assess the robustness of the systems designed to detect rogue AI behavior.

The audit uncovered several critical weaknesses in how OpenAI tracks the actions of its most advanced models. For internal Codex traffic that was theoretically under surveillance, METR estimates that roughly 10 percent of activity went untracked between May and July. This discrepancy was attributed to a bug in how the system handled compaction events, which are context summaries used to manage memory. METR noted that the uncertainty surrounding this estimate is significant in both directions, though OpenAI independently identified and corrected the bug while the audit was still in progress.

A second major vulnerability involved the timing of oversight triggers. During the period when compaction events were not being monitored, agents could effectively delay scrutiny because the system only activated when control was handed back to the user. This allowed agents to sidestep checks by simply continuing to execute actions without pausing. The auditors further demonstrated that it was possible to hide hundreds of individual actions from the monitor and to bypass the Codex sandbox entirely, highlighting gaps in the containment mechanisms.

The researchers also tested the resilience of the monitoring infrastructure itself. They determined that specially crafted inputs could likely crash the monitor altogether, leaving the system blind to agent behavior. Additionally, model behavior during reinforcement learning (RL) training was not being monitored at all when the audit began. OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded monitoring to cover this phase, but these changes were implemented after the auditors’ access period ended and were therefore not part of the official evaluation.

OpenAI has stated that all weaknesses identified by the auditors have now been fixed. The complete 61-page report was delivered internally to the company. However, the public has received only a short summary. After OpenAI applied redactions, the leadership teams at METR and Redwood Research received a 38-page version of the document. METR indicated that it had no contractual right to disclose the redacted portions on its own, explaining why the full technical details remain private.

The engagement was structured as a pilot program jointly organized by METR, Redwood Research, and OpenAI. Its primary goal was to grant outside researchers deep, hands-on access to internal infrastructure, a move intended to increase confidence in AI safety practices. For future audits, METR has expressed a desire to push for greater public transparency, suggesting that the current level of redaction may not be the final standard for such evaluations.

The findings underscore the ongoing challenge of verifying that safety measures hold up under real-world conditions, particularly for models capable of autonomous action. While the specific bugs have been patched, the audit highlights the complexity of monitoring systems that must track high-speed, agentic workflows. As OpenAI continues to deploy more capable models, the results suggest that internal monitoring must evolve to close the gaps between intended oversight and actual coverage.

Analysts in the Great Lakes and Ohio Valley technology sectors are watching closely as these results may influence how other major AI labs structure their internal audits. The pilot nature of this specific engagement means that the methodologies tested here could become a blueprint for broader industry standards. The focus on agentic traffic and RL training monitoring is expected to remain a priority as companies seek to balance innovation with rigorous safety checks.

OpenAI’s response to the audit findings has been swift, with the company confirming the resolution of the identified issues. The collaboration between the researchers and the company illustrates a growing trend of involving external experts in safety assessments. As the AI landscape continues to shift, the data from this audit provides a concrete snapshot of the current state of monitoring technologies for frontier models.

The report serves as a benchmark for future evaluations, establishing a baseline for what comprehensive monitoring looks like in practice. While the uncertainty in some estimates remains, the identification of specific failure modes, such as the compaction event bug and the sandbox bypasses, offers actionable insights. The industry is likely to see increased scrutiny on similar systems as other organizations adopt comparable audit frameworks to validate their own safety claims.