Audio Question Answering (AQA) constitutes a pivotal task in which machines
analyze both audio signals and natural language questions to produce precise
natural language answers. The significance of possessing high-quality, diverse,
and extensive AQA datasets cannot be overstated when aiming for the precision
of an AQA system. While there has been notable focus on developing accurate and
efficient AQA models, the creation of high-quality, diverse, and extensive
datasets for the specific task at hand has not garnered considerable attention.
To address this challenge, this work makes several contributions. We introduce
a scalable AQA data generation pipeline, denoted as the AQUALLM framework,
which relies on Large Language Models (LLMs). This framework utilizes existing
audio-caption annotations and incorporates state-of-the-art LLMs to generate
expansive, high-quality AQA datasets. Additionally, we present three extensive
and high-quality benchmark datasets for AQA, contributing significantly to the
progression of AQA research. AQA models trained on the proposed datasets set
superior benchmarks compared to the existing state-of-the-art. Moreover, models
trained on our datasets demonstrate enhanced generalizability when compared to
models trained using human-annotated AQA data. Code and datasets will be
accessible on GitHub~footnote{url{https://github.com/swarupbehera/AQUALLM}}.
Audio Question Answering (AQA) is a challenging task in which AI systems analyze both audio signals and natural language questions to generate accurate natural language answers. To ensure the precision of AQA systems, it is crucial to have high-quality, diverse, and extensive datasets specifically tailored for AQA. However, the creation of such datasets has not received much attention compared to the development of accurate AQA models.
This work addresses this challenge by introducing the AQUALLM framework, a scalable AQA data generation pipeline. This framework leverages Large Language Models (LLMs) and utilizes existing audio-caption annotations to generate expansive and high-quality AQA datasets. By incorporating state-of-the-art LLMs, the AQUALLM framework can produce datasets that significantly contribute to the progression of AQA research.
In addition to the framework, this work also presents three benchmark datasets for AQA. These datasets are extensive and of high quality, raising the bar for AQA research. AQA models trained on these datasets outperform existing state-of-the-art models, demonstrating their superiority. Furthermore, models trained using the proposed datasets show enhanced generalizability in comparison to models trained on human-annotated AQA data.
The multi-disciplinary nature of this work is evident in its use of both audio signal analysis and natural language processing techniques. By combining these disciplines, the AQUALLM framework enables the generation of comprehensive AQA datasets that capture the complexities of audio understanding and question answering.
This work also has significant implications for multimedia information systems. With the proliferation of audio content in various domains, such as podcasts, voice assistants, and audio recordings, the ability to extract information and provide accurate answers from audio becomes increasingly important. AQA systems built upon the datasets and frameworks presented here can greatly enhance the capabilities of multimedia information systems.
Furthermore, this work aligns with the fields of Animations, Artificial Reality, Augmented Reality, and Virtual Realities (AR/VR). Given the immersive nature of AR/VR experiences, the ability to interact with audio-based content becomes crucial. AQA systems that can understand and answer audio questions provide users with a more immersive and interactive AR/VR experience.
In conclusion, this article highlights the importance of high-quality AQA datasets and introduces the AQUALLM framework for generating such datasets. The benchmark datasets presented here raise the bar for AQA research and demonstrate the potential for models trained on these datasets to outperform existing state-of-the-art models. The multi-disciplinary nature of this work and its relevance to multimedia information systems, Animations, Artificial Reality, Augmented Reality, and Virtual Realities make it a significant contribution to the field.
Code and datasets: Accessible on GitHub: https://github.com/swarupbehera/AQUALLM