Offline-to-online latent preference transfer

PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

PrefO2O learns a task-invariant latent preference space from a small set of offline source-task preferences, then transfers reward priors and confidence-weighted pseudo-preferences to unseen target tasks without collecting target-task human feedback.

Ruiqi Wang¹, Dezhong Zhao², Xuesu Xiao³, Erdem Biyik¹, Byung-Cheol Min⁴, Guohua Chen²

¹University of Southern California ²Beijing University of Chemical Technology ³George Mason University ⁴Indiana University

Under review

Paper BibTeX

Conceptual overview of PrefO2O latent preference transfer — Source-task preferences are distilled into a shared preference manifold, then queried online to guide unseen target tasks through transferable priors and pseudo-preference supervision.

0 target-task labels

100 source labels per task

8 target benchmarks

7/10 real Cube Stacking

Motivation

Preference feedback should not restart from zero for every robot task.

Preference-based RL aligns robot behavior with human intent, but most methods still require fresh preference labels whenever the task changes. That per-task annotation cost becomes a bottleneck for practical robotic deployment.

PrefO2O treats preference as a transferable structure. Instead of aligning raw trajectories, it learns a latent space where source and target trajectory pairs can be compared by shared behavioral qualities such as progress, directness, smoothness, and stable interaction.

Method

Learn the preference manifold offline, reuse it online.

PrefO2O combines a trajectory encoder, preference VAE, and meta reward model in the source domain, then performs OOD-aware optimal transport in the learned preference space during target-task learning.

Offline latent preference learning

Source-task trajectory pairs are encoded into behavioral representations. A preference VAE shapes a task-invariant latent manifold, while a meta reward model learns a reusable preference-aligned reward prior.

Online zero-shot transfer

On an unseen target task, PrefO2O warm-starts the reward model and encoder, freezes the preference VAE, aligns source and target latent pairs with optimal transport, and propagates confidence-weighted labels.

Reward adaptation and policy optimization

Pseudo-preference labels are filtered by OOD-aware curriculum sampling and confidence weighting, keeping target reward learning stable as the policy explores new trajectories.

Detailed PrefO2O offline and online framework — Offline, PrefO2O jointly trains the encoder, preference VAE, and meta reward model. Online, the frozen preference manifold provides stable OT correspondences for label propagation and confidence-weighted reward learning on unlabeled target tasks.

Representation analysis

The latent space mixes domains while preserving preference labels.

The learned space is more task- and embodiment-invariant than raw trajectory-pair representations, yet it remains organized around interpretable preference-relevant behavior.

t-SNE visualization of raw and latent preference spaces — t-SNE views show stronger cross-task mixing and clearer preference-label separation in the learned latent space than in the raw trajectory-pair space.

Semantic attributes aligned with the learned preference axis — Projection along the learned preference axis correlates with behavioral attributes such as approach progress, path directness, and motion smoothness, including under cross-embodiment transfer.

Results

Zero target labels, competitive policy learning.

Across Meta-World, Robosuite, and DeepMind Control Suite, PrefO2O consistently outperforms preference transfer baselines and is competitive with oracle RL and PbRL references trained with privileged supervision.

83.9% Meta-World pseudo-label accuracy in intra-embodiment transfer

83.8% DMControl pseudo-label accuracy in intra-embodiment transfer

64.7% Meta-World pseudo-label accuracy retained across embodiments

5-15 pp typical performance drop under embodiment shift

Learning curves comparing PrefO2O against baselines — PrefO2O improves policy learning across same-embodiment and cross-embodiment transfer protocols, outperforming PEARL, PrefFew, and RL-VLM-F while approaching oracle references on several tasks.

Pseudo-label accuracy comparison across benchmarks — OT in the learned latent preference space produces higher-quality pseudo-labels than raw-space transfer or VLM-generated feedback, especially under cross-embodiment shift.

Ablation study for PrefO2O — Removing the latent preference space causes the most consistent degradation; OOD sampling and weighting matter most when the target policy visits sparse regions of the preference manifold.

One-to-one transfer

Real-world robot validation

Simulation-trained policies transfer to a physical Kinova Jaco2 arm.

PrefO2O policies are trained in simulation, then deployed on real Cube Stacking and Pick-and-Place tasks without collecting target-task human feedback on the physical robot.

Cube Stacking

7/10 real-world success with PrefO2O.

Pick-and-Place

6/10 real-world success with PrefO2O.

Citation

BibTeX

@article{wang2026prefo2o,
  title   = {PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer},
  author  = {Wang, Ruiqi and Zhao, Dezhong and Xiao, Xuesu and Biyik, Erdem and Min, Byung-Cheol and Chen, Guohua},
  journal = {Under review},
  year    = {2026}
}