What AI Learns from Our Judgments

A conversation with Tom Sühr on the preferences that shape Artificial Intelligence

Longer answers, more flattery, more bullet points: what we like in an AI response shapes how the next model answers. Tom Sühr researches the interaction between humans and AI; he explores how human feedback shapes machine learning, and why this doesn't always result in better answers.

When we use ChatGPT and similar tools, we don't usually consider whether the AI is learning from us. Are we all unwittingly helping to develop AI?

Interactions with users generate valuable data for AI providers and, depending on the company, may be used in model development—unless you opt out in the settings. In the race between providers, anything that makes the models better gets put to use.

At what points does human feedback come into AI development?

I consider two stages of training of language models at which it occurs. The first is alignment, where the model learns to recognise a good answer. This encompasses not only facts, but also format, length, writing style, and tone. The standard technique, which has been used for a long time, is Reinforcement Learning from Human Feedback. Many people will be familiar with it: ChatGPT shows you two answers and asks you to choose the better one. This information can be used to train future models.

And the second point at which human feedback comes in?

That is the evaluation process; it uses the same method. Specialized companies offer platforms where you receive two answers from different models. The one that wins the most often is considered the best. This results in influential rankings for research and industry.

Depending on who you ask, opinions will likely vary widely.

That’s right. And by choosing who to survey, you determine whose preferences the model learns. For example, US users might have different style and length preferences to Asian users.

But beyond that, there is another challenge: The quality of a text has many dimensions—is the content accurate, is it easy to read, and is the style appropriate? A simple assessment like “Text A is better” hardly does it justice, and that can lead to problems. Some issues have already been fairly well researched: if the training is too intensive, the texts become extremely long, as longer texts apparently win these comparisons more often. There is also excessive use of formatting, such as bullet points and emojis. Added to this is sycophancy: the model constantly praises and flatters users: “Great question!”, “Thanks for the excellent point!” It is not yet clear whether this is also an artefact of pairwise comparisons.

How do you study questions like these?

To explore these questions, we developed Comparity.ai, a freely available platform where several thousand users can test various AI models and select the better of two answers in each case. With existing datasets, we only know the outcome: model, prompt, response. Only in an experiment of our own can we trace what's actually going on. We want to understand effects such as excessive length and sycophancy: how do they arise, and how can they be prevented? We’re interested in the criteria people use to make their choices and what impact each has on the AI. To find out, we also train our own models on the data we collect and compare them.

Can you give any tips for not being dazzled by the most impressive-looking AI answer?

The models are simply extremely good, and it's hard to stay critical of them. Unfortunately, I can't offer a solution yet, but we're working on it. Ultimately, it's like mushroom picking: the biggest, most magnificent mushroom isn't always the most palatable. The real question, first and foremost, is what criteria we use to make our decisions. Our data suggests that length tempts people to click, while the unassuming, short answers tend to get left behind. But in the end, a small mushroom can sometimes be the better one.