Why AI language models struggle to “see” where atoms sit in a material

In materials science, an increasingly common approach is to write out a crystal structure as text: its elements, the dimensions of its repeating unit and the coordinates of every atom. An LLM is then trained on this text to predict properties such as formation energy, which indicates how stable a material is, or stiffness.

A crystal is defined by two kinds of information: which elements it contains, and where exactly its atoms sit in three-dimensional space. Nawaf Alampara and Kevin Maik Jablonka of Friedrich Schiller University Jena (Helmholtz AI Associate) and Santiago Miret of Intel Labs tested how well LLMs use each. The study, “Less can be more for predicting properties with large language models”, is published in Nature Computational Science.

Separating “what” from “where”

The team took two approaches. First, they created synthetic test data in which they could decide how much a property depends on atom positions. Each value came from a simple, physics-inspired energy formula with two parts. One part is based only on which atoms are present and how many of each. The other is based on the distances between them. By changing the weight of each part, the team could see whether a model’s accuracy dropped as positions became more important.

Second, they built MatText, an open software framework that turns crystal structures into nine different text formats. These range from a plain chemical formula to full files listing the position of every atom. The team used it to test small models pretrained on up to two million crystal structures, as well as general-purpose LLMs with up to 70 billion parameters.

“Geometric blindness”

In the synthetic tests, the language models learned much more from the types of atoms than from their positions. They behaved almost exactly like n-gram models, a far simpler method that only looks at short sequences of text. The real benchmark tasks showed the same thing. Text formats with no atomic coordinates at all, such as lists of which atoms are bonded or descriptions of each atom’s local environment, often did as well as or better than full coordinate files. This is what the paper’s title, “Less can be more”, refers to.

The researchers call this “geometric blindness”. Within the range they tested, neither larger models nor more training data made a noticeable difference, and the problem appeared in every text format they tried. Changing how numbers are split into pieces for the model to read did not consistently help either. According to the authors, language models would need more fundamental changes to their architecture before they can make good use of atomic positions.

Choosing the right tool

Published results on MatBench, a widely used materials benchmark, point the same way. Graph neural networks (GNNs), which represent a crystal as a network of connected atoms, consistently beat approaches based on language models. The gap is so clear that the authors call it the “GNN-LM wall”.

LLMs are not without use, though. They learn well from which elements a material contains and from descriptions of each atom’s surroundings. The authors therefore suggest using language models where a property depends mainly on composition, and geometric models where it depends on spatial structure. More generally, they argue that researchers should understand the assumptions built into a model, known as its “inductive biases”, before using it for a scientific prediction problem.

The MatText code is available on GitHub and the datasets on Hugging Face, so other researchers can test their own models against the same benchmark.