by jsendak | Jul 24, 2024 | Computer Science
The rapid advancement of foundation models in medical imaging is a promising development that has the potential to greatly enhance diagnostic accuracy and personalized treatment in healthcare. However, incorporating these models into medical practice requires careful consideration of their trustworthiness. Trustworthiness encompasses various aspects including privacy, robustness, reliability, explainability, and fairness. In order to fully assess the trustworthiness of foundation models, it is important to conduct thorough examinations and evaluations.
While there is a growing body of literature on foundation models in medical imaging, there are significant gaps in knowledge, particularly in the area of trustworthiness. Existing surveys on trustworthiness tend to overlook the specific variations and applications of foundation models within the medical imaging domain. This survey paper aims to address these gaps by reviewing current research on foundation models in major medical imaging applications such as segmentation, medical report generation, medical question and answering (Q&A), and disease diagnosis. The focus of these reviews is on papers that explicitly discuss trustworthiness.
It is important to explore the challenges associated with making foundation models trustworthy in each specific application. For example, in segmentation tasks, trustworthiness can be compromised if the model fails to accurately identify and classify the different regions of an image. Similarly, in medical report generation, errors or biases in the model’s predictions can undermine trust. Ensuring trustworthiness in medical Q&A and disease diagnosis is also crucial, as incorrect or unreliable answers can have serious consequences for patient care.
The authors of this survey paper summarize the current concerns and strategies for enhancing trustworthiness in foundation models for medical image analysis. They also highlight the future promises of these models in revolutionizing patient care. It is clear that trustworthiness is a critical factor in the successful deployment of these models in healthcare, and there is a need for a balanced approach that fosters innovation while maintaining ethical and equitable healthcare delivery. Advances in trustworthiness evaluation methods, transparency in model development, and standardized guidelines can all contribute to achieving trustworthy AI in medical image analysis.
Key Takeaways:
- The deployment of foundation models in healthcare requires a rigorous examination of their trustworthiness.
- Existing surveys on foundation models in medical imaging lack focus on trustworthiness and fail to address specific variations and applications.
- This survey paper reviews research on foundation models in major medical imaging applications, emphasizing trustworthiness discussions.
- Challenges in making foundation models trustworthy vary across applications such as segmentation, medical report generation, Q&A, and disease diagnosis.
- The paper highlights current concerns, strategies, and future promises of foundation models in revolutionizing patient care.
- A balanced approach is necessary to foster innovation while ensuring ethical and equitable healthcare delivery.
In conclusion, the survey paper emphasizes the importance of trustworthiness in foundation models for medical imaging. Addressing the gaps in existing literature and exploring the challenges and strategies associated with trustworthiness will contribute to the advancement of trustworthy AI in healthcare. The potential benefits of these models in improving diagnostic accuracy and personalized treatment are substantial, but it is essential to prioritize the ethical and equitable delivery of healthcare in their development and deployment.
Read the original article
by jsendak | Jul 23, 2024 | Computer Science
arXiv:2407.14936v1 Announce Type: new
Abstract: Brain-computer interface (BCI) facilitates direct communication between the human brain and external systems by utilizing brain signals, eliminating the need for conventional communication methods such as speaking, writing, or typing. Nevertheless, the continuous generation of brain signals in BCI frameworks poses challenges for efficient storage and real-time transmission. While considering the human brain as a semantic source, the meaningful information associated with cognitive activities often gets obscured by substantial noise present in acquired brain signals, resulting in abundant redundancy. In this paper, we propose a cross-modal brain-computer semantic communication paradigm, named EidetiCom, for decoding visual perception under limited-bandwidth constraint. The framework consists of three hierarchical layers, each responsible for compressing the semantic information of brain signals into representative features. These low-dimensional compact features are transmitted and converted into semantically meaningful representations at the receiver side, serving three distinct tasks for decoding visual perception: brain signal-based visual classification, brain-to-caption translation, and brain-to-image generation, in a scalable manner. Through extensive qualitative and quantitative experiments, we demonstrate that the proposed paradigm facilitates the semantic communication under low bit rate conditions ranging from 0.017 to 0.192 bits-per-sample, achieving high-quality semantic reconstruction and highlighting its potential for efficient storage and real-time communication of brain recordings in BCI applications, such as eidetic memory storage and assistive communication for patients.
Decoding Visual Perception through Brain-Computer Semantic Communication
The field of Brain-Computer Interfaces (BCIs) has made significant strides in facilitating direct communication between the human brain and external systems. This article introduces a novel approach called EidetiCom, which leverages cross-modal brain-computer semantic communication to decode visual perception under limited-bandwidth constraint.
BCIs typically involve the acquisition and analysis of brain signals to interpret the user’s intentions or cognitive activities. However, the continuous generation of brain signals poses challenges in terms of efficient storage and real-time transmission. The authors of this paper recognize that the meaningful information associated with cognitive activities often gets obscured by noise, resulting in redundancy.
EidetiCom addresses this challenge by proposing a three-layer hierarchical framework. Each layer is responsible for compressing the semantic information of brain signals into representative features. These low-dimensional compact features are then transmitted and converted into semantically meaningful representations at the receiving end. This approach enables three distinct tasks for decoding visual perception: brain signal-based visual classification, brain-to-caption translation, and brain-to-image generation.
The multi-disciplinary nature of this concept is evident in its integration of brain signals, visual perception, and semantic communication. By combining knowledge from fields such as neuroscience, computer vision, and data compression, EidetiCom presents a holistic solution for efficient storage and real-time communication of brain recordings.
From a multimedia information systems perspective, EidetiCom bridges the gap between brain signals and visual perception. By decoding and reconstructing visual information from brain signals, it enables the creation of virtual realities and augmented realities that can be experienced by users. This has significant implications for fields such as gaming, virtual reality simulations, and assistive communication for patients.
The utilization of EidetiCom in BCI applications, such as eidetic memory storage, holds promise for personalized memory augmentation and retrieval. Additionally, its potential for assistive communication can empower individuals with speech or motor disabilities to communicate effectively.
In conclusion, the proposed cross-modal brain-computer semantic communication paradigm, EidetiCom, demonstrates its ability to facilitate semantic communication under low bit rate conditions. With its focus on efficient storage and real-time transmission of brain recordings, EidetiCom paves the way for advancements in multimedia information systems, animations, artificial reality, augmented reality, and virtual realities.
Read the original article
by jsendak | Jul 23, 2024 | Computer Science
Expert Commentary: Advancements in Edge Device Deep Learning
Recent advancements in research and technology have paved the way for on-device computation of deep learning tasks, bringing advanced AI capabilities to edge devices and micro-controller units (MCUs). This has opened up new possibilities for deploying deep neural net (DNN) models on battery-less intermittent devices, which were once constrained by limited power and resources.
One of the key approaches in enabling deep learning on edge devices is through the optimization of DNN models. This involves techniques such as weight sharing, pruning, and neural architecture search (NAS) to tailor the models for specific edge devices. By reducing the model size and optimizing its architecture, these techniques make it possible to run DNN models on devices with limited resources, such as those with SRAM under 256KB.
However, previous optimization techniques did not take into account intermittent execution or power constraints during NAS. They primarily focused on consecutive execution without power loss, and intermittent execution designs only considered data reuse and costs related to intermittent inference, often resulting in low accuracy. This limitation led to the need for a new approach that could harness the power of optimized DNN models specifically targeting SRAM under 256KB and make them schedulable and runnable within intermittent power.
Accelerated Intermittent Deep Inference: Overcoming Limitations
Our research team has proposed a novel solution called Accelerated Intermittent Deep Inference, which addresses the limitations of previous approaches. Our main contributions are:
- Scheduling tasks performed by on-device inferencing into intermittent execution cycles and optimizing for latency.
- Developing a system that can achieve end-to-end latency while maintaining higher accuracy compared to existing baseline models optimized for edge devices.
By carefully scheduling the execution of deep inference tasks within intermittent execution cycles, we are able to utilize the available power more efficiently and minimize latency. This is crucial for achieving real-time responsiveness on edge devices while running resource-intensive DNN models.
In addition to efficient scheduling, we have also developed a system that takes into account the intermittent nature of power availability. By optimizing DNN models specifically for SRAM under 256KB and designing the system to handle intermittent execution, we are able to achieve a much higher accuracy compared to previous approaches.
The Accelerated Intermittent Deep Inference approach not only overcomes the limitations of existing techniques but also opens up new possibilities for deploying deep learning on battery-less intermittent devices. This has tremendous implications for various applications, including IoT devices, wearables, and edge computing.
Overall, the advancements in edge device deep learning are promising, and the proposed Accelerated Intermittent Deep Inference approach presents a significant breakthrough. By optimizing DNN models and designing systems that can handle intermittent execution, we are able to bring high-accuracy deep learning capabilities to resource-constrained edge devices. This will fuel further innovation in AI and enable a wide range of applications in the IoT and edge computing domains.
Read the original article
by jsendak | Jul 22, 2024 | Computer Science
arXiv:2407.14093v1 Announce Type: new
Abstract: Recently, mixture of experts (MoE) has become a popular paradigm for achieving the trade-off between modal capacity and efficiency of multi-modal large language models (MLLMs). Different from previous efforts, we are dedicated to exploring the dynamic expert path in an already exist MLLM and show that a standard MLLM can be also a mixture of experts. To approach this target, we propose a novel dynamic expert scheme for MLLMs, termed Routing Experts (RoE), which can achieve example-dependent optimal path routing without obvious structure tweaks. Meanwhile, a new regularization of structure sparsity is also introduced to enforce MLLMs to learn more short-cut inference, ensuring the efficiency. In addition, we also realize the first attempt of aligning the training and inference schemes of MLLMs in terms of network routing. To validate RoE, we apply it to a set of latest MLLMs, including LLaVA-1.5, LLaVA-HR and VILA, and conduct extensive experiments on a bunch of VL benchmarks. The experiment results not only show the great advantages of our RoE in improving MLLMs’ efficiency, but also yield obvious advantages than MoE-LLaVA in both performance and speed, e.g., an average performance gain of 3.3% on 5 benchmarks while being faster.
Exploring the Dynamic Expert Path in Multi-Modal Large Language Models
In recent years, the use of multi-modal large language models (MLLMs) has gained popularity in various applications such as natural language processing, computer vision, and information retrieval. These models combine different modalities (e.g., text, images, audio) to achieve better performance. However, one of the challenges in MLLMs is finding the right balance between model capacity and efficiency.
A new approach called mixture of experts (MoE) has emerged as a solution to this challenge. MoE allows for the combination of multiple modalities while efficiently utilizing computational resources. The concept of MoE involves dividing the model into multiple “experts” that specialize in processing specific modalities. These experts then collaborate to make predictions.
In this article, the authors propose a novel approach called Routing Experts (RoE) to further enhance the efficiency of MLLMs. Unlike previous approaches, RoE focuses on dynamically routing examples to the most appropriate expert, without the need for significant modifications to the model structure. This dynamic routing allows for example-dependent optimal path routing, leading to improved performance.
Additionally, the authors introduce a new regularization technique to enforce structure sparsity in MLLMs. This regularization encourages the learning of more efficient inference pathways within the models, further enhancing efficiency. The authors also highlight the significance of aligning the training and inference schemes of MLLMs, ensuring consistency in network routing.
To validate the effectiveness of RoE, the authors conduct extensive experiments on a set of state-of-the-art MLLMs, including LLaVA-1.5, LLaVA-HR, and VILA. These models are evaluated on a range of visual-language benchmarks. The experimental results demonstrate that RoE not only improves the efficiency of MLLMs but also outperforms MoE-LLaVA in terms of both performance and speed. On average, RoE achieves a 3.3% performance gain across five benchmarks while being faster.
This research highlights the multi-disciplinary nature of the concepts involved. The combination of natural language processing, computer vision, and neural networks makes this work relevant to the wider field of multimedia information systems. The concepts of RoE and MoE can also be extended to other areas such as animations, artificial reality, augmented reality, and virtual realities. By optimizing efficiency and performance in MLLMs, these concepts contribute to the development of more powerful and responsive multimedia systems.
Read the original article
by jsendak | Jul 22, 2024 | Computer Science
This study proposes a hierarchical multistakeholder vehicle-to-grid (V2G) coordination strategy that addresses the challenges surrounding renewable energy utilization, grid stability, and the optimization of benefits for all stakeholders involved. The strategy is based on safe multi-agent constrained deep reinforcement learning (MCDRL) and the Proof-of-Stake algorithm.
One of the key stakeholders in this strategy is the distribution system operator (DSO). The DSO’s primary concern is load fluctuations and the integration of renewable energy into the grid. With the increasing adoption of electric vehicles, the demand for electricity is expected to surge. By implementing the proposed strategy, the DSO can better manage these load fluctuations and leverage the flexibility offered by EVs to integrate more renewable energy into the grid.
Electric vehicle aggregators (EVAs) are another vital stakeholder in this coordination strategy. EVAs face challenges related to energy constraints and charging costs. By participating in the V2G system, EVAs can efficiently manage the energy demands of electric vehicles under their aggregation and optimize charging schedules to minimize costs.
In order for electric vehicles to participate in V2G, three critical parameters must be considered: battery conditioning, state of charge (SOC), state of power (SOP), and state of health (SOH). These parameters play a crucial role in the performance and lifespan of the EV’s battery. By considering these parameters in the coordination strategy, the study ensures that the participation of electric vehicles in V2G is sustainable and minimizes battery degradation.
The proposed hierarchical multistakeholder V2G coordination strategy offers several benefits. Firstly, it significantly enhances the integration of renewable energy into the power grid, thereby reducing reliance on conventional fossil fuels and contributing to a more sustainable energy mix. Secondly, it mitigates load fluctuations, making the power grid more resilient and reliable. Thirdly, it meets the energy demands of the EVAs, ensuring a stable and cost-efficient operation of their electric vehicle fleets. Lastly, by optimizing charging schedules and considering battery conditioning, SOC, SOP, and SOH, the strategy reduces charging costs and minimizes battery degradation, promoting the long-term viability of V2G systems.
In conclusion, the proposed hierarchical multistakeholder V2G coordination strategy based on safe multi-agent constrained deep reinforcement learning and the Proof-of-Stake algorithm is a promising approach to optimize the benefits for all stakeholders in the electric vehicle ecosystem. By addressing the challenges associated with renewable energy utilization, load fluctuations, energy constraints, and battery degradation, this strategy paves the way for a more sustainable and efficient integration of electric vehicles into the power grid.
Read the original article
by jsendak | Jul 19, 2024 | Computer Science
arXiv:2407.13111v1 Announce Type: new
Abstract: Vision foundation models are increasingly employed in autonomous driving systems due to their advanced capabilities. However, these models are susceptible to adversarial attacks, posing significant risks to the reliability and safety of autonomous vehicles. Adversaries can exploit these vulnerabilities to manipulate the vehicle’s perception of its surroundings, leading to erroneous decisions and potentially catastrophic consequences. To address this challenge, we propose a novel Precision-Guided Adversarial Attack (PG-Attack) framework that combines two techniques: Precision Mask Perturbation Attack (PMP-Attack) and Deceptive Text Patch Attack (DTP-Attack). PMP-Attack precisely targets the attack region to minimize the overall perturbation while maximizing its impact on the target object’s representation in the model’s feature space. DTP-Attack introduces deceptive text patches that disrupt the model’s understanding of the scene, further enhancing the attack’s effectiveness. Our experiments demonstrate that PG-Attack successfully deceives a variety of advanced multi-modal large models, including GPT-4V, Qwen-VL, and imp-V1. Additionally, we won First-Place in the CVPR 2024 Workshop Challenge: Black-box Adversarial Attacks on Vision Foundation Models and codes are available at https://github.com/fuhaha824/PG-Attack.
Analyzing the Precision-Guided Adversarial Attack (PG-Attack) Framework
The article introduces a novel framework, called the Precision-Guided Adversarial Attack (PG-Attack), which is aimed at addressing the vulnerabilities of vision foundation models in autonomous driving systems. These models are known to be susceptible to adversarial attacks, which can lead to incorrect perception of the vehicle’s surroundings and potentially dangerous outcomes. The PG-Attack framework combines two techniques, namely Precision Mask Perturbation Attack (PMP-Attack) and Deceptive Text Patch Attack (DTP-Attack), to deceive advanced multi-modal large models.
One of the key aspects of the PG-Attack framework is its multi-disciplinary nature. It incorporates techniques from computer vision, natural language processing, and adversarial machine learning. By combining these disciplines, the framework is able to effectively manipulate the perception of autonomous vehicles, highlighting the interconnectedness of different domains in developing advanced systems.
The PMP-Attack technique is designed to precisely target the attack region while minimizing the overall perturbation. This is important as it allows the attack to be more stealthy and less likely to be detected by the model. By focusing on specific regions, the attacker can maximize the impact on the target object’s representation in the model’s feature space, leading to more convincing deceptive inputs.
The DTP-Attack introduces deceptive text patches to disrupt the model’s understanding of the scene. This technique leverages natural language processing to generate text that is strategically placed to confuse the model. By incorporating textual information into the attack, the framework enhances its effectiveness in fooling the vision foundation models.
The experiments conducted by the authors demonstrate the success of the PG-Attack framework in deceiving various advanced multi-modal large models, including GPT-4V, Qwen-VL, and imp-V1. These models are widely used in the field of multimedia information systems, animations, artificial reality, augmented reality, and virtual realities. Therefore, the implications of these adversarial attacks are significant for the wider field.
This research highlights the need for robust defenses against adversarial attacks in autonomous driving systems. It also emphasizes the importance of considering multi-disciplinary approaches to address the vulnerabilities of complex machine learning models. The availability of the PG-Attack framework’s code on GitHub allows researchers and practitioners to study and develop countermeasures against such attacks, contributing to the overall safety and reliability of autonomous vehicles.
Read the original article
