ChatGPT Outperforms Professional Advice Columnists, Study Shows AI Gives Better Advice

A new study shows that ChatGPT's advice is perceived as more balanced compared to the responses provided by advice columnists.

The study, which was recently published in Frontiers of Psychology, has been conducted by researchers at the University of Melbourne and the University of Western Australia.

As part of the study, researchers compared ChatGPT and human responses to fifty social dilemma questions that were randomly selected from 10 popular advice columns.

When do people prefer AI?

Researchers used the paid version of ChatGPT with OpenAI's most capable large language model (LLM) GPT-4 for the study. In a pre-registered survey, they showed 404 participants a question along with the corresponding response by an advice columnist and by ChatGPT.

Next, they asked the participants to rate which answer, in their view, was more balanced, empathetic, comprehensive, helpful and generally better. The researchers found that ChatGPT outperformed the human advisers on the 5 randomly asked questions and in all categories queried.

Notably, the preference rates ranged from about 70 to 85 per cent in favour of ChatGPT. Moreover, the study indicated the ChatGPT responses were longer than those provided by the advice columnists. However, participants were unable to determine which response was written by ChatGPT.

In a second pre-registered survey, researchers constrained each ChatGPT answer to be about the same length that of the advice columnist. Interestingly, the second study corroborated the above findings. This is a major sign that the participants did not prefer ChatGPT answers solely because they were more detailed.

Most people want a human to answer their social dilemma questions

Despite the high quality of advice by ChatGPT, 77 per cent of participants said they would prefer a human answering their social dilemma questions rather than a computer. This preference for human responses is also seen in the previous study.

Since participants were not informed which responses were written by ChatGPT and which were written by humans, it is safe to say their preference for human responses does not reflect the quality of the responses. Instead, it represents a social or cultural phenomenon.

The researchers want upcoming studies to further explore this phenomenon by informing participants in advance which answers were written by humans and which were written by AI. It will be interesting to see whether this will increase their willingness to seek advice from AI-powered tools like ChatGPT.

Last month, a study showed ChatGPT can understand and respond to human emotions, which could be a sign that the AI chatbot is a step closer to artificial general intelligence (AGI). Understandably, OpenAI, the company behind ChatGPT, is offering $10 million (about £7 million) in research grants to control such superintelligent AI systems.