Separating Language From Thought To Understand Why AI Chat Bots Make Mistakes

Using linguistic and cognitive approaches, scientists from The University of Texas at Austin, Massachusetts Institute of Technology, and University of California Los Angeles propose an explanation for why AI programs like ChatGPT, which can produce fluid and coherent sentences, are sometimes prone to errors that human writers are not.

Large language models (LLMs), of which ChatGPT is one, are trained on enormous language datasets and generate text by predicting the word most likely to appear next in a sequence, not unlike the autocomplete function in email and text messaging. The resulting prose often sounds so convincingly human that readers wonder if something akin to human thinking lies behind it. But LLM-produced text also regularly contains bizarre mistakes and falsehoods. In a paper posted to the open-access archive arXiv, the authors argue that the strengths and weaknesses of LLMs can be understood by separating language performance into two aspects: formal and functional linguistic competence.

Formal linguistic competence is the command of the rules and patterns of a particular language, things like vocabulary and grammar. Functional linguistic competence, on the other hand, encompasses the abilities necessary to understand language, such as reasoning, knowledge of the world, and social cognition. Using evidence from cognitive neuroscience, the authors demonstrate that formal linguistic competence in humans arises from language-specific mechanisms in the frontal and temporal lobes of the brain, whereas functional linguistic competence relies on multiple other brain regions related to human thinking. Due to their strictly linguistic training, LLMs succeed at language formally but not always functionally. They are good at composing sentences, but not always good at thinking.

The authors describe how several fallacies that conflate language abilities and thinking abilities result in public misunderstanding of what LLMs can and cannot do. Some human observers believe that the skillful language produced by LLMs indicates an underlying sentience, or looming sentience. Others focus on the logical gaffs and argue that LLMs are thus poor language generators.

“The models' success at formal linguistic competence can make it challenging to evaluate their functional competence,” says co-lead author Kyle Mahowald assistant professor of linguistics at The University of Texas at Austin. “For almost all human history, when we see fluent writing, we assume it was written by a human with thoughts and feelings. And so we can assume something about the thought process behind that writing, based on what we know about humans. Now when we see fluent writing, it might be computer-generated. And our minds just aren't used to that.”

Ultimately, the paper argues that currently available LLMs have made great strides in acquiring formal linguistic competence but that employing methods that also address non-linguistic cognitive skills will be necessary to create programs with functional linguistic competence.

Mahowald notes that studying LLMs may also have implications for our understanding of human thinking and language.

“As a cognitive scientist and linguist, my real interest is in figuring out if large language models can tell us something about human language and human cognition,” he says. “While the models are black boxes in that we can't easily reason about why they do what they do, the human brain is even more of a black box. With LLMs, we at least know how they are trained, what their architecture is like, and so on. Understanding the similarities and differences is a huge and exciting challenge for scientists that I think could shed light on longstanding questions in linguistics, cognitive science, and psychology.”

Publication: Kyle Mahowald, et al., Dissociating language and thought in large language models: a cognitive perspective, Cornell University (2023) DOI: 10.48550/arXiv.2301.06627

Original Story Source: The University of Texas at Austin