Finding specific objects and switching between targets requires the flexible integration of visual and task-specific information. To investigate how the brain solves a target-switching task, we recorded neural responses in perirhinal cortex (PRH) and its primary input, inferotemporal cortex (IT) as subjects performed a task that required finding different targets in different blocks of trials. Our results suggest that visual and target information are initially combined within or before IT in a “tangled” or nonlinearly separable manner, followed by “untangling” computations in PRH that reformat this information such that it is more accessible via a linear read-out. These computations culminate in a subpopulation of “bingo neurons” in PRH that fire whenever a target is currently in view, across changes in target identity.