Grant: Discovery Horizons

Our project is to build and experiment with Pre-trained Language Models (PLMs) grounded to human-like inductive biases, from data annotated for patterns reflecting neurocognitive pattern biases, for improved PLM performance and increased scientific understanding. That is, we will use what we know about language and the brain to engineer better language-using computer systems. We join the recent efforts to move PLM research closer to human performance by biasing Machine Learning (ML) 'attention' in sync with human inductive biases (e.g., Linzen 2020; Rytting & Wingate 2021; Kumar et al. 2022, 2023). "Human-like inductive biases will improve [a PLM’s] ability to learn language structure and new tasks from limited data,” as Linzen puts it, “and will align the models’ generalization behavior more closely with human expectations." The engineering aim is to reduce system behavior that “is unpredictable and inconsistent with that of humans". The scientific aim is to probe what such a grounded PLM can uncover about the interconnections between neurocognition and language (e.g., Chomsky et al. 2023; Pavlick 2023; Trott et al. 2023), especially as such models align with the dominant psychological theory in linguistics, usage-based exemplar systems (e.g., Christiansen et al. 2008; Bybee 2013; Díaz-Campos & Balasch 2023). We will develop a language-focussed game (think Wordle but as a game with a purpose) to generate annotated ML training data, removing what Kumar et al. call “the main bottleneck” in this field (2023, 10). The data will be used so ‘small’ PLMs can emulate processes characteristic of human thought and language use. The annotations mark patterns that recruit human attention, bolster human memory, and even measurably affect human credibility judgements (e.g., McGlone & Tofighbakhsh 1999, 2000; Kara-Yakoubian et al 2021). The engineering payoff is that smaller PLMs avoid a welter of problems associated with very large PLMs, like ChatGPT (Bender & Koller 2020). Current PLM research in human-like inductive biases does not rely, as ours will, on what is known about human thought. Rather, it attempts to work from the bottom up, with raw data. We range from skeptical to agnostic about whether that approach will work, but we are confident that directly including patterns known to affect human attention and memory in the training data, and utilizing human judgment, in what we call a ‘middle-out approach’ (Harris et al. 2017), will be more successful.