AllenAI’s StateBench separates state tracking into four levels, from retrieving initial values to reassignment, cumulative updates and references between variables. Even frontier models fell to 13% accuracy or less on the longest, hardest inputs. In separate direct-training experiments, recurrent and hybrid models generalized some tasks to much longer sequences, distinguishing training effects from architectural ones. The benchmark fixes the number of variables and does not cover real conversations that continually introduce new entities, so test the needed tracking ability rather than relying on context-window size.
Back to Tech
StateBench evaluates long-document state tracking at four complexity levels
A long context window alone does not ensure accurate change tracking; task training and architecture matter.