Acoustic Modeling

A sharable software repository for Japanese LVCSR (Large Vocabulary Continuous Speech Recognition) is introduced. It is designed as a baseline platform for research and developed by researchers of different academic institutes under a governmental support. The repository consists of a recognition engine (Julius), Japanese acoustic models and statistical language models as well as Japanese morphological analysis tools. These modules can be easily integrated and replaced under a plug-and-play framework, which makes it possible to fairly evaluate components and to develop specific application systems. Assessment of these modules and systems in a 20000-word dictation task is reported. The software repository is freely available to the public.

Real-time multilingual interaction during mobile video calls still difficult to achieve due to strict latency, fluctuating network conditions, and the limited resources capacity of handheld devices. Although recent speech translation systems show high linguistic accuracy, many studies highly depend on cloud-based pipelines or text-only captions, which disrupt conversational flow and increase deployment cost. We design and implement a lightweight speech-to-speech translation system integrated directly into live video calls. Our system combines ondevice speech recognition and speech synthesis with a lightweight online translation service, coordinated via parallel pipeline that operates independently of the core audio-video stream. To avoid degradation of call quality, translated content is exchanged using a dedicated real-time data channel rather than the audio channel itself. Experiments conducted on physical Android smartphones under realistic network conditions over Wi-Fi and mobile data networks show end-to-end translation latency of approximately 0.5 to 2.0 seconds across multiple language pairs. Subjective user evaluations yield a mean opinion score of 4.2, indicating that the translated speech is sufficiently clear and natural for everyday conversational use.

Acoustic Modeling in today's emotion recognition engines employs general models independent of the spoken phonetic content. This seems to work well enough given sufficient instances to cover for a broad variety of phonetic structures and emotions at the same time. However, data is usually sparse in the field and the question arises whether unit specific models as word emotion models could outperform the typical general models. In this respect this paper tries to answer the question how strongly acoustic emotion models depend on the textual and phonetic content. We investigate the influence on the turn and word level by use of state-of-the-art techniques for frame and word modeling on the well-known public Berlin Emotional Speech and Speech Under Simulated and Actual Stress databases. In the result it is clearly shown that the phonetic structure does strongly influence the accuracy of emotion recognition.

Nonverbal vocalizations are one of the characteristics of spontaneous speech distinguishing it from written text. These phenomena are sometimes regarded as a problem in language and acoustic modeling. However, vocalizations such as filled pauses enhance language models at the local level and serve some additional functions (marking linguistic boundaries, signaling hesitation). In this paper we investigate a wider range of nonverbals and investigate their potential for language modeling of conversational speech, and compare different modeling approaches. We find that all nonverbal sounds, with the exception of breath, have little effect on the overall results. Due to its specific nature, as well as its frequency in the data, modeling of breath as a regular language model event leads to a substantial improvement in both perplexity and speech recognition accuracy.

In the search for a standard unit for use in recognition of emotion in speech, a whole turn, that is the full section of speech by one person in a conversation, is common. Within applications such turns often seem favorable. Yet, high effectiveness of sub-turn entities is known. In this respect a two-stage approach is investigated to provide higher temporal resolution by chunking of speech-turns according to acoustic properties, and multi-instance learning for turnmapping after individual chunk analysis. For chunking fast pre-segmentation into emotionally quasi-stationary segments by one-pass Viterbi beam search with token passing basing on MFCC is used. Chunk analysis is realized by brute-force large feature space construction with subsequent subset selection, SVM classification, and speaker normalization. Extensive tests reveal differences compared to one-stage processing. Alternatively, syllables are used for chunking.

Recognition of emotion in speech usually uses acoustic models that ignore the spoken content. Likewise one general model per emotion is trained independent of the phonetic structure. Given sufficient data, this approach seemingly works well enough. Yet, this paper tries to answer the question whether acoustic emotion recognition strongly depends on phonetic content, and if models tailored for the spoken unit can lead to higher accuracies. We therefore investigate phoneme-, and word-models by use of a large prosodic, spectral, and voice quality feature space and Support Vector Machines (SVM). Experiments also take the necessity of ASR into account to select appropriate unitmodels. Test-runs on the well-known EMO-DB database facing speaker-independence demonstrate superiority of word emotion models over today's common general models provided sufficient occurrences in the training corpus.

This study has introduced the design of a Hidden Markov Model based LVCSR system in a new target language based on a different source language and without the need of a large speech databases on the target language. The Tigrinya LVCSR was developed using an Amharic Corpus consisting of 10,850 sentences and a limited Tigrinya data containing 600 sentences to train the acoustic models. The study was conducted based on the v

The quality of seismic images obtained by reverse time migration ͑RTM͒ strongly depends on the imaging condition. We propose a new imaging condition that is motivated by stationary phase analysis of the classical crosscorrelation imaging condition. Its implementation requires the Poynting vector of the source and receiver wavefields at the imaging point. An obliquity correction is added to compensate for the reflector dip effect on amplitudes of RTM. Numerical experiments show that using an imaging condition with obliquity compensation improves reverse time migration by reducing backscattering artifacts and improving illumination compensation.

We investigate and compare several techniques for automatic recognition of unconstrained context-independent phoneme strings from TIMIT and NTIMIT databases. Among the compared techniques, the technique based on TempoRAl Patterns (TRAP) achieves the best results in the clean speech, it achieves about 10% relative improovements against baseline system. Its advantage is also observed in the presence of mismatch between training and testing conditions. Issues such as the optimal length of temporal patterns in the TRAP technique and the effectiveness of mean and variance normalization of the patterns and the multi-band input the TRAP estimations, are also explored.

Phoneme Recognizers followed by Language Modeling (PRLM) have consistently yielded top performance in language identification (LID) task. Parallel ordering of PRLMs (PPRLM) improves performance even more. Since tokenizer is the most important part of LID system the high quality phoneme recognizer is employed. Two different multilingual databases for training phoneme recognizers are compared and the amount of sufficient training data is studied. Reported results are on data from NIST 2003 LID evaluation. Our four PRLM systems have Equal Error Rate (EER) of 2.4% on 12 languages task. This result compares favorably to the best known result from this task.

Although research has previously been done on multilingual speech recognition, it has been found to be very difficult to improve over separately trained systems. The usual approach has been to use some kind of "universal phone set" that covers multiple languages. We report experiments on a different approach to multilingual speech recognition, in which the phone sets are entirely distinct but the model has parameters not tied to specific states that are shared across languages. We use a model called a "Subspace Gaussian Mixture Model" where states' distributions are Gaussian Mixture Models with a common structure, constrained to lie in a subspace of the total parameter space. The parameters that define this subspace can be shared across languages. We obtain substantial WER improvements with this approach, especially with very small amounts of inlanguage training data.

We describe an acoustic modeling approach in which all phonetic states share a common Gaussian Mixture Model structure, and the means and mixture weights vary in a subspace of the total parameter space. We call this a Subspace Gaussian Mixture Model (SGMM). Globally shared parameters define the subspace. This style of acoustic model allows for a much more compact representation and gives better results than a conventional modeling approach, particularly with smaller amounts of training data.

We describe the design of Kaldi, a free, open-source toolkit for speech recognition research. Kaldi provides a speech recognition system based on finite-state transducers (using the freely available OpenFst), together with detailed documentation and scripts for building complete recognition systems. Kaldi is written is C++, and the core library supports modeling of arbitrary phonetic-context sizes, acoustic modeling with subspace Gaussian mixture models (SGMM) as well as standard Gaussian mixture models, together with all commonly used linear and affine transforms. Kaldi is released under the Apache License v2.0, which is highly nonrestrictive, making it suitable for a wide community of users.

The AMADEUS (ANTARES Modules for the Acoustic Detection Under the Sea) system which is described in this article aims at the investigation of techniques for acoustic detection of neutrinos in the deep sea. It is integrated into the ANTARES neutrino telescope in the Mediterranean Sea. Its acoustic sensors, installed at water depths between 2050 and 2300 m, employ piezo-electric elements for the broad-band recording of signals with frequencies ranging up to 125 kHz. The typical sensitivity of the sensors is around -145 dB re 1V/µPa (including preamplifier). Completed in May 2008, AMADEUS consists of six "acoustic clusters", each comprising six acoustic sensors that are arranged at distances of roughly 1 m from each other. Two vertical mechanical structures (so-called lines) of the ANTARES detector host three acoustic clusters each. Spacings between the clusters range from 14.5 to 340 m. Each cluster contains custom-designed electronics boards to amplify and digitise the acoustic signals from the sensors. An on-shore computer cluster is used to process and filter the data stream and store the selected events. The daily volume of recorded data is about 10 GB. The system is operating continuously and automatically, requiring only little human intervention. AMADEUS allows for extensive studies of both transient signals and ambient noise in the deep sea, as well as signal correlations on several length scales and localisation of acoustic point sources. Thus the system is excellently suited to assess the background conditions for the measurement of the bipolar pulses expected to originate from neutrino interactions.

This paper deals with binaural sound localization. An active strategy is proposed, relying on a precise model of the dynamic changes induced by motion on the auditive perception. The proposed framework allows motions of both the sound source and the sensor. The resulting stochastic discretetime model is then exploited together with Unscented Kalman filtering to provide range and azimuth estimation. Simulations and experiments show the effectiveness of the method.

The aim of this research is to develop a speech synthesis model tailored towards Nigerian languages by leveraging natural language processing tool such as FastSpeech 2 and meta-tts for high-quality, non-autoregressive text-to-speech (TTS) generation and HiFi-GAN for neural vocoding. It was motivated due to lack of high-quality synthetic speech models for low-resource languages especially Nigerian Languages and specificially Hausa, Igbo and Yoruba. The methodology adopted is a structured and iterative approach that integrates Structured System Analysis and Design Methodology (SSADM), and Machine Learning Development Lifecycle (MLDLC), which incorporates a feasibility study, corpus collection, phonetic analysis, and model training with Nigerian Language annotated speech dataset. Speech dataset will be collected and preprocess from selected Nigerian languages(Igbo Hausa and Yoruba). Phonemes will be developed using both rule-based approach and grapheme-to-phoneme models like Epitran and Phonemizer, while FastSpeech 2, meta-tts and HiFi-GAN will be fine-tuned to accommodate tonal variations and prosodic patterns inherent in these languages. The model training pipeline will integrate Tacotron-based aligners for efficient text-to-mel-spectrogram conversion, while HiFi-GAN will enhance naturalness and intelligibility. Python will be the primary programming language for the implementation of this research while the interface will be a combination of hypertext markup language (HTML), cascading style sheet (CSS) and Javascript. The expected outcome is a state-of-the-art, speech synthesis system capable of generating natural and intelligible speech across multiple Nigerian languages.