RLCR Increases Our Ability to Trust AI
Rudina Seseri
At the beginning of last year, I discussed a method of making AI more truthful called Factuality-Aware Alignment for Large Language Models (FLAME). Since then, there has been a wave of new techniques and methods that improve upon that foundation of augmenting AI accuracy. Recently, a breakthrough training method was introduced called Reinforcement Learning with Calibration Rewards (RLCR) in a paper from researchers at MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). This promising new method unlocks potential new use cases for AI in industries previously weary of its accuracy and better triaging for current AI use cases.
RLCR seeks to look at a different portion of the generation process compared to FLAME. By forcing the model to reflect on the accuracy of its output, RLCR enables a higher quality model to be produced compared to standard reinforcement learning. Additionally, RLCR allows for more efficient human verification processes on tasks performed by the model, eventually unlocking faster and lower cost processes.
🗺️ What is RLCR?
RLCR is a training approach that asks models to test their confidence of answers by having it produce a probabilistic prediction score to the question “how accurate do I think my prediction is?” The model outputs this score with its answer to a given prompt. For example, if a model is asked “Where is Pennsylvania?”, the model could respond with “Mexico” and “90% confidence.” In this case, the model clearly overestimated its accuracy. Since RLCR implements the probabilistic prediction score, known as the Brier method, the model will be penalized for overestimating its confidence. The same penalty would occur if the model said the “United States” and “10% confidence.”
This prediction score is different from how traditional reinforcement learning works, where a binary answer of “correct” or “incorrect” is used to train the model. The CSAIL researchers found that the binary can make the calibration (e.g., how accurate the model tells the user it is) worse off than when the model uses RLCR.
Beyond the implications for the model’s training, RLCR unlocks an added value in the business world. Instead of a model outputting a wrong answer and receiving the feedback that it is wrong, with RLCR, teams can start to leverage answers that may be close to completely accurate or escalate to humans to ensure optimal accuracy.
🤔 Why does RLCR matter, and what are its limitations?
One of the largest problems today with AI models is that they are confidently wrong. With RLCR, users now know when to better trust models and their outputs. Given that reinforcement learning is a costly and laborious process, RLCR unlocks the ability for a user to know when to take a model’s output and run with it and when to not.
- More efficient triaging: RLCR enables better routing of issues to a company’s workforce. If a model produces something with high confidence, it requires fewer reviews by an organization’s team.  If the model is only able to produce a result with low confidence, these tasks can be routed to a human or a more powerful, costlier model.
- Higher trust in agent handoffs: Given that we are in the multi-agent age, an issue for this stage is when agents compound initial errors produced. RLCR enables the next agent in a chain to stop and ask the previous agent to reverify its work, limiting the chain reaction of an initial error.
- Better learning: According to the CSAIL researchers’ paper, RLCR makes models more likely to be trained better than traditional reinforcement learning because RLCR unlocks significantly better calibration.
That said, RLCR still has limitations that still limit our ability to trust AI models more extensively:
- Imprecise error recognition: RLCR gives a confidence number after a task is complete and does not tell the model which part of the solution is the problem.
- Ambiguity issues: RLCR works well for tasks that have a right or wrong answer. For more subjective tasks, this method is much more difficult to apply.
- Required training: Even though the model is now forcing itself to tell the user how accurate its predictions are, it does not automatically make the model smarter. If a model is often inaccurate, it will consistently tell you it is inaccurate.
🛠️ Applications of RLCR
RLCR’s applications are a best fit for environments where precision estimates help to guide processes and action to responses:
- Customer service: The customer can first interact with an agent and if an answer falls below a certain accuracy level, the issue can be escalated to a team member for an organization.
- Financial services, law, and healthcare: The CSAIL researchers specifically highlighted these use cases because these fields require high level of fidelity, which RLCR helps to estimate.
- Data operations: RLCR helps to better process large data sets such as by categorizing products and flagging issues with the data to team members through confidence level estimates.
Stay up-to-date on the latest AI news by subscribing to Rudina’s AI Atlas.