A person listening to music can be said to “hear space” in two senses: metaphorically, when the musical features of a composition, such as melody, harmony or rhythm, evoke a space (e.g., if she hears a “rising” melodic line) or suggest abstract concepts related to an imaginary spatial scene; and literally, when she perceives spatial information relating to sound sources and the spatial region where they are located. In this paper I will analyze the way we perceive space when listening to music, focusing on the experience of physical space. I will apply a model of the way we experience non-musical (or environmental) sound sources to the perception of musical sound sources. I will argue that our experience of musical and non-musical sound sources involve similar perception of the spatial properties of sounds sources, by applying a specific model of spatial experience of non-musical sound sources to some musical compositions which exemplify different roles of space as an aesthetic tool, I will argue that our experience of musical and non-musical sound sources involve similar perception of the spatial properties of sound sources. Accordingly I will suggest that if we want to capture the difference between musical experience and the auditory experience of non-musical sounds with relation to space, we should probably focus on the experience of metaphorical space.