What is the neural basis of abstraction? Working in collaboration with experimental groups who have trained monkeys and humans on a decision task, we analyze how subjects make use of visual information and feedback to infer a hidden rule, where the rule switches in an uncued fashion. We fit a suite of behavioral models and learn that while humans are close to optimal Bayesian agents, monkey behavior is better fit as reinforcement learning, with a novel additional factor included. We use this behavioral model to elucidate structure in neural activity recorded from 200 sites across the brain, finding low-dimensional and dynamic representations of stimulus, feedback and reward prediction error that support the notion of internal states captured by the behavioral model.