Unlike visual or tactile objects, the location of an auditory object is not represented on the primary sensory surface – the basilar membrane in the cochlea of the inner ear. Rather, it is computed from information converging from each ear onto neurons in the central nervous system, a process known as binaural (two-eared) hearing. Binaural hearing, particularly the ability to exploit sensitivity to small differences in the timing of sound at the two ears (interaural time differences, ITDs), underpins the ability to localize sound sources. Psychophysical studies have demonstrated human accuracy for ITDs in the range of just a few 10’s of microseconds, accuracy that has to be considered with respect to the 1-2 milliseconds duration of nerve action potentials conveying sensory information in the brain. Despite the existence of a well-established model of ITD-sensitivity, however, recent physiological and psychophysical data have called into question the existence of an explicit place code for ITD – i.e. a code in which neurons tuned to different ITDs, and by virtue of this, different auditory spatial locations, represent ITD in the form of ‘space map’. Here, I demonstrate that the code for ITD in the human brain is consistent with the neural representations reported for a range of mammalian species. Data from functional brain imaging, electro-encephalography and psychophysics indicate that a restricted range of ITD detectors is likely employed to generate a rate code for ITD. The outcome of this restricted range of ITD detectors is that auditory spatial sensitivity is best understood as showing an ‘acoustic fovea’ with neural mechanisms constrained to code best sound sources in frontal space.