Spot the Difference: Can AI Generate Plausible Christmas BMJ Titles?

Artificial intelligence (AI) technology can generate plausible, entertaining, and scientifically interesting titles for potential research articles, a University of Bristol-led study in the Christmas issue of The BMJ has found.

A study of The BMJ’s most popular Christmas research articles - which combined evidence-based science with light-hearted or quirky themes - found that AI generated titles were as attractive to readers but that, as in other areas of medicine, performance was enhanced by human input. 

As such, the researchers from the University of Bristol and Bristol Royal Hospital for Children suggest AI could have a role in generating hypotheses or directions for future research.

AI is already used to help doctors diagnose conditions, based on the idea that computer systems can learn from data and identify patterns. But can AI be used to generate worthwhile hypotheses for medical research?

To find out, the researchers used the titles of The BMJ’s 13 most read Christmas research articles of the past ten years to prompt similar AI generated titles, which they scored for scientific merit, entertainment, and plausibility. 

The ten highest and ten lowest scoring AI generated titles were then combined with ten real Christmas research articles and were rated by a random sample of 25 doctors from a range of specialties in Africa, Australia, and Europe. 

The results show that AI generated titles were rated at least as enjoyable (64% v 69%) and attractive (70% v 68%) as real titles, although the real titles were rated as more plausible (73% v 48%).

The AI generated titles overall were rated as having less scientific or educational merit than the real titles (58% v 39%), however this difference became non-significant when humans curated the AI output (58% v 49%).

This finding fits with previous work on AI suggesting that the best results come from combining machine learning with human oversight.

Of the AI generated titles, the most plausible was “The association between belief in conspiracy theories and the willingness to receive vaccinations,” and the highest rated was “The effects of free gourmet coffee on emergency department waiting times: an observational study.”

Dr Robin Marlow, Honorary Senior Lecturer in Bristol’s Centre for Academic Child Health and corresponding author, said: “The Christmas BMJ issue features unusual studies that are real but with a sideways look at things.  I wanted to find out if AI would be able to create titles for research studies that people would think are real.

“Our findings highlight the potential use of AI in clinical medicine, as decision support rather than as outright replacement of clinicians.  However, while AI is helpful, it’s not in a position to replace people quite yet.”

The researchers acknowledge some limitations, but suggest even in the context of quirky titles such as those that appear in the Christmas issues of The BMJ, AI has the potential to generate plausible outputs that are engaging and could attract potential readers.

Publication: Robin Marlow, et al., Ghost in the machine or monkey with a typewriter—generating titles for Christmas research articles in The BMJ using artificial intelligence: observational study, BMJ (2023). DOI: 10.1136/bmj-2021-067732.

Original Story Source:  University of Bristol