---
title: Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
url: https://www.ml-quant.com/papers/arxiv/2405.21046/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-09-26
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2405.21046
source_url: https://arxiv.org/abs/2405.21046
featured: 2024-06-05
citations: 115
topic: LLMs & Text
---


# Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

The Exploratory Preference Optimization (XPO) algorithm has been introduced for online exploration in Reinforcement Learning from Human Feedback (RLHF), potentially enhancing language model training.

- Source: https://arxiv.org/abs/2405.21046
- Identifier: arXiv:2405.21046
- Released: 2024-05-31
- First featured: Quant Letter No. 52 (2024-06-05): https://www.ml-quant.com/issues/2024-06-05/
- Citations (Semantic Scholar): 115
- Published in: not yet
- Topic: LLMs & Text

## Related

- [Training Language Models to Self-Correct via Reinforcement Learning](https://www.ml-quant.com/papers/arxiv/2409.12917/): SCoRe, a new online reinforcement learning approach, enhances the self-correction ability of large language models, showing top performance with Gemini 1.0 Pro and 1.5 Flash models.
- [From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function](https://www.ml-quant.com/papers/arxiv/2404.12358/): The research explores Direct Preference Optimization (DPO) in Reinforcement Learning From Human Feedback (RLHF), showing its ability to assign credit and its similarity to search-based algorithms in language generation.
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://www.ml-quant.com/papers/arxiv/2501.09686/): The article discusses advancements in Large Language Models (LLMs) reasoning, emphasizing the use of reinforcement learning and thought simulation for complex reasoning, and the potential of scaling during training and testing.
- [Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF](https://www.ml-quant.com/papers/arxiv/2405.19320/): The study presents a unified approach to reinforcement learning from human feedback for large language models, offering theoretical guarantees and practical effectiveness.
- [Can large language models explore in-context?](https://www.ml-quant.com/papers/arxiv/2403.15371/): Large Language Models such as GPT-3.5, GPT-4, and Llama2 struggle to explore in reinforcement learning environments without significant interventions, indicating the need for algorithmic interventions in complex decision-making scenarios.
- [RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs](https://www.ml-quant.com/papers/arxiv/2409.04421/): Reinforcement Learning from Prediction Feedback (RLPF) is a method that refines Large Language Models to produce concise, human-readable summaries, enhancing task performance and summary quality.