The Complaint Is the Curriculum: How Perplexity Trains Its Agent on Real Sessions

Part of the Dev Tools for PMs series.

A user asks an AI agent "How do I find my W-3 on Paychex?" - and the model confidently explains how to find a W-2 instead. That wrong-form session is the opening of Perplexity's research post "Learning from Real-World Experience," and this episode unpacks it. How do you train an AI product on real usage without imitating its mistakes? Perplexity's answer: judge turns, not sessions - imitate the good steps, correct the avoidable ones with hint-guided self-distillation, and trace complaints back to their true root cause. We walk the full pipeline (privacy filters, paired LLM judges, hindsight-bias checks, the training pass) and the refreshingly honest evaluations: tool errors fell 21% relative in live A/B tests, while user dissatisfaction didn't measurably move. What it means for any PM building a learning loop.