OpenAI’s new “CriticGPT” model is trained to criticize GPT-4 outputs

On Thursday, OpenAI researchers unveiled CriticGPT, a brand new AI mannequin designed to determine errors in code generated by ChatGPT. It goals to reinforce the method of creating AI methods behave in methods people need (referred to as “alignment”) by Reinforcement Learning from Human Feedback (RLHF), which helps human reviewers make massive language mannequin (LLM) outputs extra correct.

As outlined in a brand new analysis paper referred to as “LLM Critics Help Catch LLM Bugs,” OpenAI created CriticGPT to behave as an AI assistant to human trainers who evaluation programming code generated by the ChatGPT AI assistant. CriticGPT—primarily based on the GPT-4 household of LLMS—analyzes the code and factors out potential errors, making it simpler for people to identify errors that may in any other case go unnoticed. The researchers skilled CriticGPT on a dataset of code samples with deliberately inserted bugs, educating it to acknowledge and flag numerous coding errors.

The event of CriticGPT concerned coaching the mannequin on numerous inputs containing intentionally inserted errors. Human trainers had been requested to change code written by ChatGPT, introducing errors after which offering instance suggestions as if they’d found these bugs. This course of allowed the mannequin to learn to determine and critique numerous kinds of coding errors.

In experiments, CriticGPT demonstrated its means to catch each inserted bugs and naturally occurring errors in ChatGPT’s output. The brand new mannequin’s critiques had been most well-liked by trainers over these generated by ChatGPT itself in 63 % of circumstances involving pure bugs (the aforementioned statistic). This desire was partly resulting from CriticGPT producing fewer unhelpful “nitpicks” and producing fewer false positives, or hallucinated issues.

The researchers additionally created a brand new method they name Pressure Sampling Beam Search (FSBS). This methodology helps CriticGPT write extra detailed evaluations of code. It lets the researchers modify how thorough CriticGPT is in on the lookout for issues, whereas additionally controlling how typically it’d make up points that do not actually exist. They’ll tweak this steadiness relying on what they want for various AI coaching duties.

Curiously, the researchers discovered that CriticGPT’s capabilities lengthen past simply code evaluation. Of their experiments, they utilized the mannequin to a subset of ChatGPT coaching knowledge that had beforehand been rated as flawless by human annotators. Surprisingly, CriticGPT recognized errors in 24 % of those circumstances—errors that had been subsequently confirmed by human reviewers. OpenAI thinks this demonstrates the mannequin’s potential to generalize to non-code duties and highlights its means to catch delicate errors that even cautious human analysis may miss.

Regardless of its promising outcomes, like all AI fashions, CriticGPT has limitations. The mannequin was skilled on comparatively quick ChatGPT solutions, which can not totally put together it for evaluating longer, extra complicated duties that future AI methods may deal with. Moreover, whereas CriticGPT reduces confabulations, it would not remove them fully, and human trainers can nonetheless make labeling errors primarily based on these false outputs.

The analysis group acknowledges that CriticGPT is handiest at figuring out errors that may be pinpointed in a single particular location throughout the code. Nevertheless, real-world errors in AI outputs can typically be unfold throughout a number of elements of a solution, presenting a problem for future iterations of the mannequin.

OpenAI plans to combine CriticGPT-like fashions into its RLHF labeling pipeline, offering its trainers with AI help. For OpenAI, it is a step towards creating higher instruments for evaluating outputs from LLM methods that could be troublesome for people to fee with out extra help. Nevertheless, the researchers warning that even with instruments like CriticGPT, extraordinarily complicated duties or responses should still show difficult for human evaluators—even these assisted by AI.