Spatialized Sound

The relevance of the research topic. The issue of the spatial dimension of an audiovisual work has not lost its relevance since the emergence of sound recording and sound engineering, that is, since the beginning of the XX century. It is the study of the historical development of sound recording of musical texts that has become the basis for the analysis of the spatial sound field of screen works. The development of modern surround sound systems opens up new opportunities for the implementation of the authorial idea for authors of audiovisual works due to the expansion of the visual and expressive capabilities of the spatial sound field, which must be taken into account in modern audiovisual production.
The purpose of the article is to perform a retrospective analysis of spatial sound solutions, starting from the period of the emergence of cinema to the current possibilities of using multi-channel systems in modern screen works, focusing on the role of sound in creating the spatial field of an audiovisual work.
The research methodology is based on a systemic approach, which involves a holistic, comprehensive formation of a picture of the development of spatial sound solutions in screen works. Retrospective analysis of the development of the spatial sound field of a screen work led to the use of a historical approach to highlight the process of development of technical sound solutions in audiovisual art and their awareness of the impact on the audiovisual industry. In addition, the article uses such general scientific methods as synthesis, generalization, explanation.
The results. It is confirmed that the spatial sound field of a screen work is an important visual and expressive means of audiovisual art, the transformations of which are associated with the development of modern technologies and the digitalization of the production of art works. The implementation of sound in cinema not only led to a revolution in editing techniques, but also dramatically changed the visual and expressive means of artists, and enhanced the emotional impact of audiovisual works on viewers.
The scientific novelty of the research is the implementation of a comprehensive retrospective analysis of spatial sound solutions of audiovisual art, which has not previously been carried out in scientific research. It is the retrospective analysis of the formation and development of the spatial sound field in screen works that allows us to understand the general trends of sound solutions in audiovisual art and to highlight the role of sound in the construction of screen images.
The practical significance. The results of the study can be used in teaching courses on the history of cinema, sound engineering and sound design of audiovisual works, as well as in the creative activities of audiovisual production practitioners.
Conclusions. Sound technologies influence the way films are created and perceived no less actively than image technologies. It is the spatial sound field of a screen work in artistic unity with the dramaturgy of the film that provides impressive results in terms of the power of its influence on the audience. Spatial sound systems and technological achievements in this area are quite actively discussed in the information field, a retrospective analysis of the development of the spatial sound field of a screen work allows us to create a holistic picture of its perception as an important visual and expressive means of audiovisual art.

The following paper describes several transformations of ultrasonic data into musical material for hemispheric loudspeaker setup. This data comes from a medical prototype for the purpose of early breast cancer detection via 3D multimodal imaging. The data is acquired in a semi-ellipsoid mesh of thousands of ultrasonic emitters and receivers surrounding the measurement object with water as medium. A hemispheric loudspeaker array can reflect this aperture design and offers the possibility of projecting this data in a direct fashion for auditory display. The ultrasounds in the megahertz range are inaudible and need to be transformed into the audible range. We here describe our investigations and methods to transfer medical ultrasound data in an artistic context and report on our insights using different sonification strategies to gain audible, musical material.

The following paper describes several transformations of ultrasonic data into musical material for hemispheric loudspeaker setup. This data comes from a medical prototype for the purpose of early breast cancer detection via 3D multimodal imaging. The data is acquired in a semi-ellipsoid mesh of thousands of ultrasonic emitters and receivers surrounding the measurement object with water as medium. A hemispheric loudspeaker array can reflect this aperture design and offers the possibility of projecting this data in a direct fashion for auditory display. The ultrasounds in the megahertz range are inaudible and need to be transformed into the audible range. We here describe our investigations and methods to transfer medical ultrasound data in an artistic context and report on our insights using different sonification strategies to gain audible, musical material.

The Virtual Audio Server (VAS) is a toolkit designed for exploring problems in the creation of realistic Virtual Sonic Environments (VSE). The architecture of VAS provides an extensible framework in which new ideas can be quickly integrated into the system and experimented with.

Preserving older pedestrians’ navigation skills in urban environments is a challenge for maintaining their quality of life. However, the maps that are usually used by older pedestrians might be unsuitable to their specificities and the existing digital aids do not consider older people’s perceptual and cognitive declines or user experience. This study presents a rich description of the navigation experience of older pedestrians either with a visual (augmented reality glasses), auditory (bone conduction headphones), or a visual and haptic (smartwatch) wearable device adapted to age-related declines. These wearable devices are compared to the navigation aid older people usually use when navigating the city (their own digital or paper map). The study, with 18 participants, measured the navigation performance and captured detailed descriptions of the users’ experience using interviews. We highlight three main phenomena which impact the quality of the user experience with the four aids: ...

Two piezoelectric pads ("1" & "2" in Fig. 1) made from flat, laminated sheets of PVDF (polyvinylidene fluoride) polymer [4] are placed at the front of the shoe sole, below the regions covered by the big and small toes. One PVDF pad ("3" in Fig. 1) is placed at the heel. PVDF has already been inserted into a set of shoes to measure footfalls for wearable computing applications [5]; the signals are several volts into a high-impedance (≥ 10 MegOhm) load, thus require only simple buffering before digitization. The two front pads will measure differential toe pressure, while the back pad will measure dynamic pressure at the heel. Although PVDF does not provide a steady-state force measurement, a dancer is usually in motion, and the most important performance features arise from dynamics. A sensor system is described for instrumenting a pair of dancing shoes in order to capture many expressive degrees of freedom and use them to drive music synthesizers and computer graphics in a real-time performance. Dynamic pressure is measured at three points in the shoe sole, as are the bend of the sole, pitch and yaw shoe angles, and translational shoe positions. Data will be transmit across a 19.2 kbaud wireless link, enabling updates at 10 msec intervals.

SpatDIF, the Spatial Sound Description Interchange Format is a lightweight , human-readable syntax for storing and transmitting spatial sound scenes, serving as an independent, cross-platform and host-independent solution for spatial sound composition. The recent update to version 0.4 of the specification introduces the ability to define and store continuous trajectories on the authoring layer in a human-readable way, as well as describing groups and source spreading. As a result, SpatDIF provides a new way to exchange higher level authoring data across authoring tools that help to preserve the artistic intent in spatial music.

A low level sonification prototype of 3D point clouds for the sensory substitution of vision by audition for the visually impaired is investigated. The aim of this work is to study which point cloud features can be understood through the sonification of raw 3D data without the extraction of high-level features through algorithms. Preliminary results show the possibility for the user to localize objects and estimate their sizes but not to understand shapes of objects.

This paper outlines the background to a research project currently underway in Malaysia that, through spectography seeks to find models that might assist in the future development of a timbral notation. Located within the music creation and performance practices of the researchers, the project has elements of interculturality which both enrich and inform the research. The authors consider the nature of a music score, the explicit and implicit information it carries, and how this impacts on the models being developed. The understandings elicited to date are not only located in music practice, but are underpinned and supported by the theoretical works of a number of recent philosophers and theorists. The overall research project is broken down into smaller discrete sub-projects which are discussed, and the findings of each sub-project are then contextualized in the wider project. These findings include a discussion of the score as artifact and the potential contained within it. The fi...

Sound spatialization is a technique used in various musical genres as well as in soundtrack production for films and video games. In this context, specialized software has been developed for the design of sound trajectories we have classified as (1) basic movements, or image schemas of spatial movement; and (2) archetypal geometric figures. Our contribution is to reach an understanding of how we perceive the movement of sound in space as a result of the interaction between an agent's or listener's sensory-motor characteristics and the morphological characteristics of the stimuli and the acoustic space where such interaction occurs. An experiment was designed involving listening to auditory stimuli and associating them with the aforementioned spatial movement categories. The results suggest that in most cases, the ability to recognize moving sound is hindered when there are no visual stimuli present. Moreover, they indicate that archetypal geometric figures are rarely perceiv...

This work involves the investigation on techniques for musical spatialization, and the development of a real time spatialization system, based on a multichannel auralization engine (AUDIENCE). The constructed system uses the Ambisonics technique, which allows the recreation of a threedimensional sound field. The system reproduces spatial music through an arrangement of loudspeakers around the listener where sound sources can assume some free positions, and also be put into motion. Results of experiments with a musical piece show that the system can be used by composers, conductors, and producers for the musical spatialization task.

Interactive movement sonification has been gaining validity as a technique for biofeedback and auditory data mining in research and development for gaming, sports, and physiotherapy. Naturally, the harvesting of kinematic data over recent years has been a function of an increased availability of more portable, high-precision sensory technologies, such as smart phones, and dynamic real time programming environments, such as Max/MSP. Whereas the overlap of motor skill coordination and acoustic events has been a staple to musical pedagogy, musicians and music engineers have been surprisingly less involved than biomechanical, electrical, and computer engineers in research efforts in these fields. Thus, this paper proposes a prototype for an accessible virtual gaming interface that uses music and pitch training as positive reinforcement in the accomplishment of target postures.

Specter combines music information retrieval (MIR) with sound spatialization to provide a simple, yet versatile environment to experiment with sound spatialization for music composition and live performance. Through various interfaces and sensors, users may position sounds at arbitrary locations and trajectories in a three-dimensional plane. The system utilizes the JythonMusic environment for symbolic music processing, music information retrieval, and live audio manipulation. It also incorporates Iannix, a 3D graphical, open-source sequencer, for realtime generation, manipulation, and storing of sound trajectory scores. Finally, through Glaser, a sound manipulation instrument, Specter renders the various sounds in space. The system architecture supports different sound spatialization techniques including Ambisonics and Vector Based Amplitude Panning. Various interfaces are discussed, including a Kinect-based sensor system, a Leap-Motion-based hand-tracking interface, and a smartphonebased OSC controller. Finally, we present Migrant, a music composition, which utilizes and demonstrates Specter's ability to combine MIR techniques with sound spatialization through inexpensive, minimal hardware.

Multi-modal capabilities can be added to a simulation system in order to enhance data comprehension. We describe a system for adding sonification capabilities to a real-time computational fluid dynamics (CFD) simulator. Our system uses Max/MSP modules to add sonic properties to CFD solutions. The enhancements described in this paper allow users to locate sound sources in a 3-D environment using stereo auditory cues to identify data features.

The Sun is a resonant cavity for very low frequency acoustic waves, and just like a musical instrument, it supports a number of oscillation modes, also commonly known as harmonics. We are able to observe these harmonics by looking at how the Sun's surface oscillates in response to them. Although this data has been studied scientifically for decades, it has only rarely been sonified. The Sonification of Solar Harmonics (SoSH) Project seeks to sonify data related to the field of helioseismology and distribute tools for others to do the same. Creative applications of this research by the authors include musical compositions, installation artwork, a short documentary, and a full-dome planetarium experience.

The modern control room is predominantly made up of visual displays, which might make monitoring different processes a cumbersome and visually cognitively demanding task. Sonification could be used to support the monitoring task. However, it is not clear how the most beneficial sonification should be designed. In this pilot study an experimental setup was developed to explore perception of different sonification approaches. A user experiment was performed to assess perception of the sonification when and where simulated data deviated most from the normal level. It was found that all sonification conditions were generally useful, regardless of the participant's musical experience, shown both in terms of objective and subjective measurements. Stereo panning of the sound was also generally experienced as helpful, but the use of different pitch might not have been experienced to contribute as much for solving the task. The findings in this pilot study will be further used to create ...

Many composers of electroacoustic music create stereo works which are intended to be relayed (or diffused) in a concert venue through loudspeakers strategically placed around the audience. The process of diffusion can provide added meaning to the experience in terms ...

This paper investigates alternative methods for multichannel sound diffusion which are based on the mapping of physical gestures to spatial location. The approach addresses issues surrounding the inappropriateness of mixing consoles as an effective interface for sound diffusion in multi-channel loudspeaker environments and instead proposes a system that allows the user to traverse all possible trajectories in a multi-loudspeaker environment.

The modern control room is predominantly made up of visual displays, which might make monitoring different processes a cumbersome and visually cognitively demanding task. Sonification could be used to support the monitoring task. However, it is not clear how the most beneficial sonification should be designed. In this pilot study an experimental setup was developed to explore perception of different sonification approaches. A user experiment was performed to assess perception of the sonification when and where simulated data deviated most from the normal level. It was found that all sonification conditions were generally useful, regardless of the participant's musical experience, shown both in terms of objective and subjective measurements. Stereo panning of the sound was also generally experienced as helpful, but the use of different pitch might not have been experienced to contribute as much for solving the task. The findings in this pilot study will be further used to create ...