On Thursday, OpenAI researchers unveiled CriticGPT, a new AI model designed to establish errors in code generated by ChatGPT. It goals to improve the method of creating AI methods behave in methods people need (referred to as “alignment”) by way of Reinforcement Learning from Human Feedback (RLHF), which helps human reviewers make massive language model (LLM) outputs extra correct.
As outlined in a new analysis paper referred to as “LLM Critics Help Catch LLM Bugs,” OpenAI created CriticGPT to act as an AI assistant to human trainers who assessment programming code generated by the ChatGPT AI assistant. CriticGPT—based mostly on the GPT-4 household of LLMS—analyzes the code and factors out potential errors, making it simpler for people to spot errors that may in any other case go unnoticed. The researchers trained CriticGPT on a dataset of code samples with deliberately inserted bugs, instructing it to acknowledge and flag numerous coding errors.
The researchers discovered that CriticGPT’s critiques have been most well-liked by annotators over human critiques in 63 p.c of circumstances involving naturally occurring LLM errors and that human-machine groups utilizing CriticGPT wrote extra complete critiques than people alone whereas lowering confabulation (hallucination) charges in contrast to AI-only critiques.
Developing an automatic critic
The growth of CriticGPT concerned coaching the model on numerous inputs containing intentionally inserted errors. Human trainers have been requested to modify code written by ChatGPT, introducing errors after which offering instance suggestions as if they’d found these bugs. This course of allowed the model to learn the way to establish and critique numerous kinds of coding errors.
In experiments, CriticGPT demonstrated its skill to catch each inserted bugs and naturally occurring errors in ChatGPT’s output. The new model’s critiques have been most well-liked by trainers over these generated by ChatGPT itself in 63 p.c of circumstances involving pure bugs (the aforementioned statistic). This desire was partly due to CriticGPT producing fewer unhelpful “nitpicks” and producing fewer false positives, or hallucinated issues.
The researchers additionally created a new approach they name Force Sampling Beam Search (FSBS). This technique helps CriticGPT write extra detailed opinions of code. It lets the researchers modify how thorough CriticGPT is in searching for issues whereas additionally controlling how typically it would make up points that do not actually exist. They can tweak this steadiness relying on what they want for various AI coaching duties.
Interestingly, the researchers discovered that CriticGPT’s capabilities lengthen past simply code assessment. In their experiments, they utilized the model to a subset of ChatGPT coaching information that had beforehand been rated as flawless by human annotators. Surprisingly, CriticGPT recognized errors in 24 p.c of those circumstances—errors that have been subsequently confirmed by human reviewers. OpenAI thinks this demonstrates the model’s potential to generalize to non-code duties and highlights its skill to catch delicate errors that even cautious human analysis would possibly miss.
Despite its promising outcomes, like all AI fashions, CriticGPT has limitations. The model was trained on comparatively brief ChatGPT solutions, which can not totally put together it for evaluating longer, extra complicated duties that future AI methods would possibly deal with. Additionally, whereas CriticGPT reduces confabulations, it would not remove them solely, and human trainers can nonetheless make labeling errors based mostly on these false outputs.
The analysis group acknowledges that CriticGPT is best at figuring out errors that may be pinpointed in a single particular location inside the code. However, real-world errors in AI outputs can typically be unfold throughout a number of components of a solution, presenting a problem for future model iterations.
OpenAI plans to combine CriticGPT-like fashions into its RLHF labeling pipeline, offering its trainers with AI help. For OpenAI, it is a step towards growing higher instruments for evaluating outputs from LLM methods that could be troublesome for people to price with out further assist. However, the researchers warning that even with instruments like CriticGPT, extraordinarily complicated duties or responses should show difficult for human evaluators—even these assisted by AI.