For many species, the ability to encode the location of a sound source relies on comparing differences in the phase of the acoustic waveform arriving at each eardrum (the interaural time difference, ITD). This comparison takes place three synaptic stages beyond the sensory epithelium of the inner ear, requiring extraordinary, and well-documented, neural mechanisms to ensure fidelity of the signal to this point. Following this stage of processing, a significant transformation in neural coding of ITD occurs in the auditory midbrain, rendering neurons sensitive not only to the instantaneous ITD, but also the longer-term context in which ITDs were presented. Using a combination of single-neuron recordings and modeling, we demonstrate that at the level of midbrain neurons, this sensitivity requires the action of both brain hemispheres and a range of neural time-constants, implemented variously through circuits implementing fast (iontotropic) and slow (inhibitory metabotropic) neurotransmitters. Psychophysical experiments support the notion that the context in which ITDs are heard influences performance in a discrimination task in which noise bursts containing ITDs are made more discriminable by appropriate (i.e. co-located) preceding sounds hypothesis, an observation that accords with the efficient coding hypothesis.