arXiv:2409.12304v1 Announce Type: new Abstract: Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that encompasses a wide variety of symptoms and degrees of impairment, which makes the diagnosis and treatment challenging. Functional magnetic resonance imaging (fMRI) has been extensively used to study brain activity in ASD, and machine learning methods have been applied to analyze resting state fMRI (rs-fMRI) data. However, fewer studies have explored the recent transformer-based models on rs-fMRI data. Given the superiority of transformer models in capturing long-range dependencies in sequence data, we have developed a transformer-based self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. To address over-fitting in small datasets and enhance the model performance, we propose self-supervised pre-training tasks to reconstruct the randomly masked fMRI time-series data, investigating the effects of various masking strategies. We then finetune the model for the ASD classification task and evaluate it using two public datasets and five-fold cross-validation with different amounts of training data. The experiments show that randomly masking entire ROIs gives better model performance than randomly masking time points in the pre-training step, resulting in an average improvement of 10.8% for AUC and 9.3% for subject accuracy compared with the transformer model trained from scratch across different levels of training data availability. Our code is available on GitHub.
This article explores the use of functional magnetic resonance imaging (fMRI) and machine learning methods to analyze resting state fMRI (rs-fMRI) data in individuals with Autism Spectrum Disorder (ASD). While previous studies have utilized fMRI to study brain activity in ASD, this study focuses on the application of transformer-based models to rs-fMRI data. The researchers have developed a self-supervised framework that directly analyzes time-series fMRI data without computing functional connectivity. To enhance model performance and address over-fitting in small datasets, the researchers propose self-supervised pre-training tasks that involve reconstructing randomly masked fMRI time-series data. The effects of various masking strategies are investigated. The model is then fine-tuned for the ASD classification task and evaluated using two public datasets and five-fold cross-validation with different amounts of training data. The results demonstrate that randomly masking entire regions of interest (ROIs) during pre-training yields better model performance compared to randomly masking time points. This approach leads to an average improvement of 10.8% for AUC (area under the curve) and 9.3% for subject accuracy compared to training the transformer model from scratch. The code for this study is available on GitHub.
Exploring Transformer-Based Models for Analyzing Resting State fMRI Data in Autism Spectrum Disorder
Autism Spectrum Disorder (ASD) is a complex neurodevelopmental condition characterized by a wide range of symptoms and varying degrees of impairment. The diagnosis and treatment of ASD pose significant challenges due to the diverse nature of the disorder. In recent years, functional magnetic resonance imaging (fMRI) has emerged as a powerful tool for studying brain activity in individuals with ASD. Additionally, machine learning methods have been successfully utilized to analyze resting state fMRI (rs-fMRI) data, providing valuable insights into the neural mechanisms underlying the disorder.
However, previous studies have primarily focused on traditional machine learning algorithms and have not fully explored the potential of transformer-based models in analyzing rs-fMRI data. Transformers, originally developed for natural language processing tasks, have demonstrated exceptional capabilities in capturing long-range dependencies in sequential data. Leveraging this advantage, we have developed a novel transformer-based self-supervised framework specifically designed for analyzing time-series fMRI data without relying on traditional functional connectivity computations.
To address the challenge of overfitting in small datasets and enhance model performance, we propose a self-supervised pre-training approach that involves reconstructing randomly masked fMRI time-series data. This approach allows the model to learn meaningful representations of the underlying brain activity patterns. We investigate the effects of various masking strategies to optimize the pre-training task.
Following the self-supervised pre-training phase, we fine-tune the model for the specific task of ASD classification. We evaluate the performance of our model using two publicly available datasets and employ a five-fold cross-validation strategy with varying amounts of training data. The experimental results demonstrate that randomly masking entire regions of interest (ROIs) during pre-training improves the model’s overall performance compared to randomly masking individual time points. On average, this approach leads to an improvement of 10.8% in area under the curve (AUC) and 9.3% in subject accuracy, when compared to training the transformer model from scratch across different levels of training data availability.
Our transformer-based framework represents a significant step in leveraging advanced deep learning techniques for the analysis of rs-fMRI data in individuals with ASD. By directly analyzing time-series fMRI data without relying on functional connectivity computations, our approach provides a more comprehensive understanding of the underlying neural mechanisms associated with ASD. The improved model performance achieved through self-supervised pre-training tasks highlights the importance of utilizing unsupervised learning methods in addressing the challenges of limited data availability.
Researchers and practitioners interested in exploring our work further can access our code on GitHub. By encouraging collaboration and open-source development, we aim to foster an environment of innovation and progress in the field of ASD research.
The paper titled “Autism Spectrum Disorder Classification using Transformer-based Self-supervised Learning on Resting State fMRI Data” presents a novel approach to analyzing resting state fMRI data for the classification of Autism Spectrum Disorder (ASD). ASD is a complex neurodevelopmental condition with a wide range of symptoms and levels of impairment, making accurate diagnosis and treatment challenging.
The authors highlight the extensive use of functional magnetic resonance imaging (fMRI) in studying brain activity in ASD, but note that fewer studies have explored the application of transformer-based models on resting state fMRI data. Transformers have shown superiority in capturing long-range dependencies in sequence data, which makes them a promising approach for analyzing fMRI time-series data.
To address the limitations of small datasets and improve model performance, the authors propose a self-supervised pre-training framework. This framework involves reconstructing randomly masked fMRI time-series data, with different masking strategies explored. The goal is to enhance the model’s ability to generalize and reduce overfitting.
The results of the experiments conducted using two public datasets and five-fold cross-validation demonstrate the effectiveness of the proposed approach. Randomly masking entire regions of interest (ROIs) during pre-training yields better model performance compared to randomly masking time points. This approach leads to an average improvement of 10.8% for area under the curve (AUC) and 9.3% for subject accuracy compared to training the transformer model from scratch.
Overall, this study contributes to the field by showcasing the potential of transformer-based models in analyzing resting state fMRI data for ASD classification. The use of self-supervised pre-training and the exploration of different masking strategies add valuable insights to the methodology. The availability of the code on GitHub further facilitates reproducibility and encourages further research in this area.
Looking ahead, future research could focus on several aspects. Firstly, expanding the evaluation to larger and more diverse datasets would strengthen the generalizability of the proposed framework. Additionally, investigating the interpretability of the transformer-based model’s predictions could provide insights into the neural correlates of ASD. Furthermore, exploring the potential of transfer learning by fine-tuning the model on related neurodevelopmental disorders could be an interesting avenue to explore. Overall, the combination of transformer-based models and self-supervised learning holds promise for advancing our understanding of ASD and potentially improving its diagnosis and treatment.
Read the original article