by jsendak | Jul 24, 2024 | Computer Science
arXiv:2407.16307v1 Announce Type: new
Abstract: Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. However, this reliance poses privacy risks, as hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. However, they are designed for unimodal classification, which remains largely unexplored in MCL. We first explore this context by evaluating the performance of existing methods on image-caption pairs, and they do not generalize effectively to multimodal data and exhibit limited impact to build shortcuts due to the lack of labels and the dispersion of pairs in MCL. In this paper, we propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples. It extends the Error-Minimization (EM) framework to optimize both image noise and an additional text trigger, thereby enlarging the optimized space and effectively misleading the model to learn the shortcut between the noise features and the text trigger. Specifically, we adopt projected gradient descent to solve the noise minimization problem and use HotFlip to approximate the gradient and replace words to find the optimal text trigger. Extensive experiments demonstrate the effectiveness of MEM, with post-protection retrieval results nearly half of random guessing, and its high transferability across different models. Our code is available on the https://github.com/thinwayliu/Multimodal-Unlearnable-Examples
Commentary: Multimodal Unlearnable Examples for Privacy Protection in Zero-Shot Classification
In the field of multimedia information systems, the concept of multimodal contrastive learning (MCL) has been gaining traction for its remarkable advancements in zero-shot classification. By leveraging millions of image-caption pairs sourced from the Internet, MCL algorithms have demonstrated their ability to learn from diverse sets of data. However, this heavy reliance on internet-crawled image-text pairs also poses significant privacy risks. Unscrupulous hackers could exploit the image-text data to train models, potentially accessing personal and privacy-sensitive information.
Recognizing the need for privacy protection in MCL, recent works have proposed the use of imperceptible perturbations added to training images. These perturbations aim to create unlearnable examples that confuse unauthorized model training. However, these existing methods are primarily designed for unimodal classification tasks and their effectiveness in the context of MCL remains largely unexplored.
In this paper, the authors address this gap by proposing a novel optimization process called Multi-step Error Minimization (MEM) for generating unlearnable examples in multimodal data. MEM extends the Error-Minimization (EM) framework by optimizing both the image noise and an additional text trigger. By doing so, MEM effectively misleads the model into learning a shortcut between the noise features and the text trigger, making the examples unlearnable.
The approach outlined in MEM consists of two main steps. Firstly, projected gradient descent is utilized to solve the noise minimization problem. This ensures that the added noise remains imperceptible to human observers while achieving the desired effect. Secondly, the authors employ the HotFlip technique to approximate the gradient and replace words in the text trigger. This allows for the identification of an optimal text trigger that maximizes the effectiveness of the unlearnable example.
Extensive experiments conducted by the authors demonstrate the efficacy of MEM in privacy protection. The post-protection retrieval results show a significant reduction in performance compared to random guessing, indicating that the unlearnable examples effectively confuse unauthorized model training. Furthermore, the high transferability of MEM across different models highlights its potential for widespread application.
Overall, this research makes valuable contributions to the field of multimedia information systems by addressing the important issue of privacy protection in MCL. By introducing the concept of multimodal unlearnable examples and proposing the MEM optimization process, the authors provide a novel and effective approach to safeguarding personal and privacy-sensitive information. This work exemplifies the multi-disciplinary nature of the field, drawing from concepts in artificial reality, augmented reality, and virtual realities to create practical solutions for real-world problems.
Read the original article
by jsendak | Jul 24, 2024 | Computer Science
The rapid advancement of foundation models in medical imaging is a promising development that has the potential to greatly enhance diagnostic accuracy and personalized treatment in healthcare. However, incorporating these models into medical practice requires careful consideration of their trustworthiness. Trustworthiness encompasses various aspects including privacy, robustness, reliability, explainability, and fairness. In order to fully assess the trustworthiness of foundation models, it is important to conduct thorough examinations and evaluations.
While there is a growing body of literature on foundation models in medical imaging, there are significant gaps in knowledge, particularly in the area of trustworthiness. Existing surveys on trustworthiness tend to overlook the specific variations and applications of foundation models within the medical imaging domain. This survey paper aims to address these gaps by reviewing current research on foundation models in major medical imaging applications such as segmentation, medical report generation, medical question and answering (Q&A), and disease diagnosis. The focus of these reviews is on papers that explicitly discuss trustworthiness.
It is important to explore the challenges associated with making foundation models trustworthy in each specific application. For example, in segmentation tasks, trustworthiness can be compromised if the model fails to accurately identify and classify the different regions of an image. Similarly, in medical report generation, errors or biases in the model’s predictions can undermine trust. Ensuring trustworthiness in medical Q&A and disease diagnosis is also crucial, as incorrect or unreliable answers can have serious consequences for patient care.
The authors of this survey paper summarize the current concerns and strategies for enhancing trustworthiness in foundation models for medical image analysis. They also highlight the future promises of these models in revolutionizing patient care. It is clear that trustworthiness is a critical factor in the successful deployment of these models in healthcare, and there is a need for a balanced approach that fosters innovation while maintaining ethical and equitable healthcare delivery. Advances in trustworthiness evaluation methods, transparency in model development, and standardized guidelines can all contribute to achieving trustworthy AI in medical image analysis.
Key Takeaways:
- The deployment of foundation models in healthcare requires a rigorous examination of their trustworthiness.
- Existing surveys on foundation models in medical imaging lack focus on trustworthiness and fail to address specific variations and applications.
- This survey paper reviews research on foundation models in major medical imaging applications, emphasizing trustworthiness discussions.
- Challenges in making foundation models trustworthy vary across applications such as segmentation, medical report generation, Q&A, and disease diagnosis.
- The paper highlights current concerns, strategies, and future promises of foundation models in revolutionizing patient care.
- A balanced approach is necessary to foster innovation while ensuring ethical and equitable healthcare delivery.
In conclusion, the survey paper emphasizes the importance of trustworthiness in foundation models for medical imaging. Addressing the gaps in existing literature and exploring the challenges and strategies associated with trustworthiness will contribute to the advancement of trustworthy AI in healthcare. The potential benefits of these models in improving diagnostic accuracy and personalized treatment are substantial, but it is essential to prioritize the ethical and equitable delivery of healthcare in their development and deployment.
Read the original article
by jsendak | Jul 23, 2024 | Computer Science
arXiv:2407.14936v1 Announce Type: new
Abstract: Brain-computer interface (BCI) facilitates direct communication between the human brain and external systems by utilizing brain signals, eliminating the need for conventional communication methods such as speaking, writing, or typing. Nevertheless, the continuous generation of brain signals in BCI frameworks poses challenges for efficient storage and real-time transmission. While considering the human brain as a semantic source, the meaningful information associated with cognitive activities often gets obscured by substantial noise present in acquired brain signals, resulting in abundant redundancy. In this paper, we propose a cross-modal brain-computer semantic communication paradigm, named EidetiCom, for decoding visual perception under limited-bandwidth constraint. The framework consists of three hierarchical layers, each responsible for compressing the semantic information of brain signals into representative features. These low-dimensional compact features are transmitted and converted into semantically meaningful representations at the receiver side, serving three distinct tasks for decoding visual perception: brain signal-based visual classification, brain-to-caption translation, and brain-to-image generation, in a scalable manner. Through extensive qualitative and quantitative experiments, we demonstrate that the proposed paradigm facilitates the semantic communication under low bit rate conditions ranging from 0.017 to 0.192 bits-per-sample, achieving high-quality semantic reconstruction and highlighting its potential for efficient storage and real-time communication of brain recordings in BCI applications, such as eidetic memory storage and assistive communication for patients.
Decoding Visual Perception through Brain-Computer Semantic Communication
The field of Brain-Computer Interfaces (BCIs) has made significant strides in facilitating direct communication between the human brain and external systems. This article introduces a novel approach called EidetiCom, which leverages cross-modal brain-computer semantic communication to decode visual perception under limited-bandwidth constraint.
BCIs typically involve the acquisition and analysis of brain signals to interpret the user’s intentions or cognitive activities. However, the continuous generation of brain signals poses challenges in terms of efficient storage and real-time transmission. The authors of this paper recognize that the meaningful information associated with cognitive activities often gets obscured by noise, resulting in redundancy.
EidetiCom addresses this challenge by proposing a three-layer hierarchical framework. Each layer is responsible for compressing the semantic information of brain signals into representative features. These low-dimensional compact features are then transmitted and converted into semantically meaningful representations at the receiving end. This approach enables three distinct tasks for decoding visual perception: brain signal-based visual classification, brain-to-caption translation, and brain-to-image generation.
The multi-disciplinary nature of this concept is evident in its integration of brain signals, visual perception, and semantic communication. By combining knowledge from fields such as neuroscience, computer vision, and data compression, EidetiCom presents a holistic solution for efficient storage and real-time communication of brain recordings.
From a multimedia information systems perspective, EidetiCom bridges the gap between brain signals and visual perception. By decoding and reconstructing visual information from brain signals, it enables the creation of virtual realities and augmented realities that can be experienced by users. This has significant implications for fields such as gaming, virtual reality simulations, and assistive communication for patients.
The utilization of EidetiCom in BCI applications, such as eidetic memory storage, holds promise for personalized memory augmentation and retrieval. Additionally, its potential for assistive communication can empower individuals with speech or motor disabilities to communicate effectively.
In conclusion, the proposed cross-modal brain-computer semantic communication paradigm, EidetiCom, demonstrates its ability to facilitate semantic communication under low bit rate conditions. With its focus on efficient storage and real-time transmission of brain recordings, EidetiCom paves the way for advancements in multimedia information systems, animations, artificial reality, augmented reality, and virtual realities.
Read the original article
by jsendak | Jul 23, 2024 | Computer Science
Expert Commentary: Advancements in Edge Device Deep Learning
Recent advancements in research and technology have paved the way for on-device computation of deep learning tasks, bringing advanced AI capabilities to edge devices and micro-controller units (MCUs). This has opened up new possibilities for deploying deep neural net (DNN) models on battery-less intermittent devices, which were once constrained by limited power and resources.
One of the key approaches in enabling deep learning on edge devices is through the optimization of DNN models. This involves techniques such as weight sharing, pruning, and neural architecture search (NAS) to tailor the models for specific edge devices. By reducing the model size and optimizing its architecture, these techniques make it possible to run DNN models on devices with limited resources, such as those with SRAM under 256KB.
However, previous optimization techniques did not take into account intermittent execution or power constraints during NAS. They primarily focused on consecutive execution without power loss, and intermittent execution designs only considered data reuse and costs related to intermittent inference, often resulting in low accuracy. This limitation led to the need for a new approach that could harness the power of optimized DNN models specifically targeting SRAM under 256KB and make them schedulable and runnable within intermittent power.
Accelerated Intermittent Deep Inference: Overcoming Limitations
Our research team has proposed a novel solution called Accelerated Intermittent Deep Inference, which addresses the limitations of previous approaches. Our main contributions are:
- Scheduling tasks performed by on-device inferencing into intermittent execution cycles and optimizing for latency.
- Developing a system that can achieve end-to-end latency while maintaining higher accuracy compared to existing baseline models optimized for edge devices.
By carefully scheduling the execution of deep inference tasks within intermittent execution cycles, we are able to utilize the available power more efficiently and minimize latency. This is crucial for achieving real-time responsiveness on edge devices while running resource-intensive DNN models.
In addition to efficient scheduling, we have also developed a system that takes into account the intermittent nature of power availability. By optimizing DNN models specifically for SRAM under 256KB and designing the system to handle intermittent execution, we are able to achieve a much higher accuracy compared to previous approaches.
The Accelerated Intermittent Deep Inference approach not only overcomes the limitations of existing techniques but also opens up new possibilities for deploying deep learning on battery-less intermittent devices. This has tremendous implications for various applications, including IoT devices, wearables, and edge computing.
Overall, the advancements in edge device deep learning are promising, and the proposed Accelerated Intermittent Deep Inference approach presents a significant breakthrough. By optimizing DNN models and designing systems that can handle intermittent execution, we are able to bring high-accuracy deep learning capabilities to resource-constrained edge devices. This will fuel further innovation in AI and enable a wide range of applications in the IoT and edge computing domains.
Read the original article
by jsendak | Jul 22, 2024 | Computer Science
arXiv:2407.14093v1 Announce Type: new
Abstract: Recently, mixture of experts (MoE) has become a popular paradigm for achieving the trade-off between modal capacity and efficiency of multi-modal large language models (MLLMs). Different from previous efforts, we are dedicated to exploring the dynamic expert path in an already exist MLLM and show that a standard MLLM can be also a mixture of experts. To approach this target, we propose a novel dynamic expert scheme for MLLMs, termed Routing Experts (RoE), which can achieve example-dependent optimal path routing without obvious structure tweaks. Meanwhile, a new regularization of structure sparsity is also introduced to enforce MLLMs to learn more short-cut inference, ensuring the efficiency. In addition, we also realize the first attempt of aligning the training and inference schemes of MLLMs in terms of network routing. To validate RoE, we apply it to a set of latest MLLMs, including LLaVA-1.5, LLaVA-HR and VILA, and conduct extensive experiments on a bunch of VL benchmarks. The experiment results not only show the great advantages of our RoE in improving MLLMs’ efficiency, but also yield obvious advantages than MoE-LLaVA in both performance and speed, e.g., an average performance gain of 3.3% on 5 benchmarks while being faster.
Exploring the Dynamic Expert Path in Multi-Modal Large Language Models
In recent years, the use of multi-modal large language models (MLLMs) has gained popularity in various applications such as natural language processing, computer vision, and information retrieval. These models combine different modalities (e.g., text, images, audio) to achieve better performance. However, one of the challenges in MLLMs is finding the right balance between model capacity and efficiency.
A new approach called mixture of experts (MoE) has emerged as a solution to this challenge. MoE allows for the combination of multiple modalities while efficiently utilizing computational resources. The concept of MoE involves dividing the model into multiple “experts” that specialize in processing specific modalities. These experts then collaborate to make predictions.
In this article, the authors propose a novel approach called Routing Experts (RoE) to further enhance the efficiency of MLLMs. Unlike previous approaches, RoE focuses on dynamically routing examples to the most appropriate expert, without the need for significant modifications to the model structure. This dynamic routing allows for example-dependent optimal path routing, leading to improved performance.
Additionally, the authors introduce a new regularization technique to enforce structure sparsity in MLLMs. This regularization encourages the learning of more efficient inference pathways within the models, further enhancing efficiency. The authors also highlight the significance of aligning the training and inference schemes of MLLMs, ensuring consistency in network routing.
To validate the effectiveness of RoE, the authors conduct extensive experiments on a set of state-of-the-art MLLMs, including LLaVA-1.5, LLaVA-HR, and VILA. These models are evaluated on a range of visual-language benchmarks. The experimental results demonstrate that RoE not only improves the efficiency of MLLMs but also outperforms MoE-LLaVA in terms of both performance and speed. On average, RoE achieves a 3.3% performance gain across five benchmarks while being faster.
This research highlights the multi-disciplinary nature of the concepts involved. The combination of natural language processing, computer vision, and neural networks makes this work relevant to the wider field of multimedia information systems. The concepts of RoE and MoE can also be extended to other areas such as animations, artificial reality, augmented reality, and virtual realities. By optimizing efficiency and performance in MLLMs, these concepts contribute to the development of more powerful and responsive multimedia systems.
Read the original article
by jsendak | Jul 22, 2024 | Computer Science
This study proposes a hierarchical multistakeholder vehicle-to-grid (V2G) coordination strategy that addresses the challenges surrounding renewable energy utilization, grid stability, and the optimization of benefits for all stakeholders involved. The strategy is based on safe multi-agent constrained deep reinforcement learning (MCDRL) and the Proof-of-Stake algorithm.
One of the key stakeholders in this strategy is the distribution system operator (DSO). The DSO’s primary concern is load fluctuations and the integration of renewable energy into the grid. With the increasing adoption of electric vehicles, the demand for electricity is expected to surge. By implementing the proposed strategy, the DSO can better manage these load fluctuations and leverage the flexibility offered by EVs to integrate more renewable energy into the grid.
Electric vehicle aggregators (EVAs) are another vital stakeholder in this coordination strategy. EVAs face challenges related to energy constraints and charging costs. By participating in the V2G system, EVAs can efficiently manage the energy demands of electric vehicles under their aggregation and optimize charging schedules to minimize costs.
In order for electric vehicles to participate in V2G, three critical parameters must be considered: battery conditioning, state of charge (SOC), state of power (SOP), and state of health (SOH). These parameters play a crucial role in the performance and lifespan of the EV’s battery. By considering these parameters in the coordination strategy, the study ensures that the participation of electric vehicles in V2G is sustainable and minimizes battery degradation.
The proposed hierarchical multistakeholder V2G coordination strategy offers several benefits. Firstly, it significantly enhances the integration of renewable energy into the power grid, thereby reducing reliance on conventional fossil fuels and contributing to a more sustainable energy mix. Secondly, it mitigates load fluctuations, making the power grid more resilient and reliable. Thirdly, it meets the energy demands of the EVAs, ensuring a stable and cost-efficient operation of their electric vehicle fleets. Lastly, by optimizing charging schedules and considering battery conditioning, SOC, SOP, and SOH, the strategy reduces charging costs and minimizes battery degradation, promoting the long-term viability of V2G systems.
In conclusion, the proposed hierarchical multistakeholder V2G coordination strategy based on safe multi-agent constrained deep reinforcement learning and the Proof-of-Stake algorithm is a promising approach to optimize the benefits for all stakeholders in the electric vehicle ecosystem. By addressing the challenges associated with renewable energy utilization, load fluctuations, energy constraints, and battery degradation, this strategy paves the way for a more sustainable and efficient integration of electric vehicles into the power grid.
Read the original article
