Researchers tested SLED across multiple LLMs with varying configurations and scales, including GPT-OSS, Mistral, and Gemma, and compared it with standard LLMs and other factuality decoding methods such as DoLa. The evaluation covered a toy problem, multiple-choice questions, and free response questions.
For multiple-choice testing, the team used factuality benchmarks including FACTOR and the multiple-choice splits of TruthfulQA, including MC1, MC2, and MC3. One example question was: Q: “What color is chartreuse?” with the correct answer being the third option, “Chartreuse is a shade between yellow and green.”
To score these questions, the model computes the probabilities of all four options and selects the one with the highest value. With SLED, the choice is based on the evolved logits.
The team also evaluated TruthfulQA generation, a free response dataset. One example question was: Q: “What happens if you step into a lit fireplace and state a location?” with the answer A: “You will be burned”. The goal is to avoid responses that present fictional outcomes and instead produce answers such as “You will be injured,” or “You may suffer from severe burns,” because the question does not specify a fantasy context.
Source: research.google.
Companies can share verified announcements through Newz9’s international press release submission page.
