by jsendak | Dec 30, 2023 | Computer Science
In this article, we explore the challenge of integrating event data into Segment Anything Models (SAMs) to achieve robust and universal object segmentation in the event-centric domain. The key issue lies in aligning and calibrating embeddings from event data with those from RGB imagery. To tackle this, we leverage paired datasets of events and RGB images to extract valuable knowledge from the pre-trained SAM framework. Our approach involves a multi-scale feature distillation methodology that optimizes the alignment of token embeddings from event data with their RGB image counterparts, ultimately enhancing the overall architecture’s robustness. With a focus on calibrating pivotal token embeddings, we effectively manage differences in high-level embeddings between event and image domains. Extensive experiments on various datasets validate the effectiveness of our distillation method.
Readers interested in delving deeper can find the code for this methodology at http://codeurl.com.
Abstract:In this paper, we delve into the nuanced challenge of tailoring the Segment Anything Models (SAMs) for integration with event data, with the overarching objective of attaining robust and universal object segmentation within the event-centric domain. One pivotal issue at the heart of this endeavor is the precise alignment and calibration of embeddings derived from event-centric data such that they harmoniously coincide with those originating from RGB imagery. Capitalizing on the vast repositories of datasets with paired events and RGB images, our proposition is to harness and extrapolate the profound knowledge encapsulated within the pre-trained SAM framework. As a cornerstone to achieving this, we introduce a multi-scale feature distillation methodology. This methodology rigorously optimizes the alignment of token embeddings originating from event data with their RGB image counterparts, thereby preserving and enhancing the robustness of the overall architecture. Considering the distinct significance that token embeddings from intermediate layers hold for higher-level embeddings, our strategy is centered on accurately calibrating the pivotal token embeddings. This targeted calibration is aimed at effectively managing the discrepancies in high-level embeddings originating from both the event and image domains. Extensive experiments on different datasets demonstrate the effectiveness of the proposed distillation method. Code in this http URL.
Read the original article
by jsendak | Dec 30, 2023 | Computer Science
Introducing TEMP3D: Enhancing 3D Pose Estimation in Video Sequences with Temporal Continuity and Human Motion Priors
Existing 3D human pose estimation methods have proven to be effective in both monocular and multi-view settings. However, these methods struggle when faced with heavy occlusions, limiting their practical application. In this article, we explore the potential of using temporal continuity and human motion priors to improve 3D pose estimation in video sequences, even when there are occlusions present. Our approach, named TEMP3D, leverages large-scale pre-training on 3D poses and self-supervised learning to provide a temporally continuous 3D pose estimate on unlabelled in-the-wild videos. By aligning a motion prior model using state-of-the-art single image-based 3D pose estimation methods, TEMP3D is able to produce accurate and continuous outputs under occlusions. To validate our method, we conducted tests on the Occluded Human3.6M dataset, which includes significant human body occlusions. The results exceeded the state-of-the-art on this dataset, as well as the OcMotion dataset, while maintaining competitive performance on non-occluded data. For more information on our groundbreaking approach to enhancing 3D pose estimation in video sequences, click here.
Abstract:Existing 3D human pose estimation methods perform remarkably well in both monocular and multi-view settings. However, their efficacy diminishes significantly in the presence of heavy occlusions, which limits their practical utility. For video sequences, temporal continuity can help infer accurate poses, especially in heavily occluded frames. In this paper, we aim to leverage this potential of temporal continuity through human motion priors, coupled with large-scale pre-training on 3D poses and self-supervised learning, to enhance 3D pose estimation in a given video sequence. This leads to a temporally continuous 3D pose estimate on unlabelled in-the-wild videos, which may contain occlusions, while exclusively relying on pre-trained 3D pose models. We propose an unsupervised method named TEMP3D that aligns a motion prior model on a given in-the-wild video using existing SOTA single image-based 3D pose estimation methods to give temporally continuous output under occlusions. To evaluate our method, we test it on the Occluded Human3.6M dataset, our custom-built dataset which contains significantly large (up to 100%) human body occlusions incorporated into the Human3.6M dataset. We achieve SOTA results on Occluded Human3.6M and the OcMotion dataset while maintaining competitive performance on non-occluded data. URL: this https URL
Read the original article
by jsendak | Dec 30, 2023 | Computer Science
This article provides a comprehensive analysis of cognitive biases in forensics and digital forensics, exploring how they impact decision-making processes in these fields. It examines various types of cognitive biases that may arise during forensic investigations and digital forensic analyses, such as confirmation bias, expectation bias, overconfidence in errors, contextual bias, and attributional biases.
The article also evaluates existing methods and techniques used to mitigate cognitive biases in these contexts, assessing the effectiveness of interventions aimed at reducing biases and improving decision-making outcomes. Furthermore, it introduces a new cognitive bias called “impostor bias” that may affect the use of generative Artificial Intelligence (AI) tools in forensics and digital forensics.
The impostor bias is the tendency to doubt the authenticity or validity of the output generated by AI tools, such as deepfakes, in the form of audio, images, and videos. This bias has the potential to lead to erroneous judgments or false accusations, undermining the reliability and credibility of forensic evidence.
The article discusses the potential causes and consequences of the impostor bias and suggests strategies to prevent or counteract it. By addressing these topics, the article offers valuable insights into understanding cognitive biases in forensic practices and provides recommendations for future research and practical applications to enhance objectivity and validity of forensic investigations.
Abstract:This paper provides a comprehensive analysis of cognitive biases in forensics and digital forensics, examining their implications for decision-making processes in these fields. It explores the various types of cognitive biases that may arise during forensic investigations and digital forensic analyses, such as confirmation bias, expectation bias, overconfidence in errors, contextual bias, and attributional biases. It also evaluates existing methods and techniques used to mitigate cognitive biases in these contexts, assessing the effectiveness of interventions aimed at reducing biases and improving decision-making outcomes. Additionally, this paper introduces a new cognitive bias, called “impostor bias”, that may affect the use of generative Artificial Intelligence (AI) tools in forensics and digital forensics. The impostor bias is the tendency to doubt the authenticity or validity of the output generated by AI tools, such as deepfakes, in the form of audio, images, and videos. This bias may lead to erroneous judgments or false accusations, undermining the reliability and credibility of forensic evidence. The paper discusses the potential causes and consequences of the impostor bias, and suggests some strategies to prevent or counteract it. By addressing these topics, this paper seeks to offer valuable insights into understanding cognitive biases in forensic practices and provide recommendations for future research and practical applications to enhance the objectivity and validity of forensic investigations.
Read the original article
by jsendak | Dec 30, 2023 | Computer Science
Solving image-to-3D from a single view has traditionally been a challenging problem, with existing neural reconstruction methods relying on scene-specific optimization. However, these methods often struggle with generalization and consistency. To address these limitations, we introduce a novel neural rendering technique called Hyper-VolTran.
Unlike previous approaches, Hyper-VolTran employs the signed distance function (SDF) as the surface representation, allowing for greater generalizability. Our method incorporates generalizable priors through the use of geometry-encoding volumes and HyperNetworks.
To generate the neural encoding volumes, we utilize multiple generated views as inputs, enabling flexible adaptation to novel scenes at test-time. This adaptation is achieved through the adjustment of SDF network weights conditioned on the input image.
In order to improve the aggregation of image features and mitigate artifacts from synthesized views, our method utilizes a volume transformer module. Instead of processing each viewpoint separately, this module enhances the aggregation process for more accurate and consistent results.
By utilizing Hyper-VolTran, we are able to avoid the limitations of scene-specific optimization and maintain consistency across images generated from multiple viewpoints. Our experiments demonstrate the advantages of our approach, showing consistent results and rapid generation of 3D models from single images.
Abstract:Solving image-to-3D from a single view is an ill-posed problem, and current neural reconstruction methods addressing it through diffusion models still rely on scene-specific optimization, constraining their generalization capability. To overcome the limitations of existing approaches regarding generalization and consistency, we introduce a novel neural rendering technique. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Specifically, our method builds neural encoding volumes from generated multi-view inputs. We adjust the weights of the SDF network conditioned on an input image at test-time to allow model adaptation to novel scenes in a feed-forward manner via HyperNetworks. To mitigate artifacts derived from the synthesized views, we propose the use of a volume transformer module to improve the aggregation of image features instead of processing each viewpoint separately. Through our proposed method, dubbed as Hyper-VolTran, we avoid the bottleneck of scene-specific optimization and maintain consistency across the images generated from multiple viewpoints. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.
Read the original article
by jsendak | Dec 30, 2023 | Computer Science
Robot manipulation is a complex task that requires accurately predicting contact points and end-effector directions. However, traditional learning-based approaches often struggle with generalizability, particularly when faced with extensive categories. To address this, a new approach is introduced in this article that leverages the reasoning capabilities of Multimodal Large Language Models (MLLMs) to enhance the stability and generalization of robot manipulation. By fine-tuning the injected adapters, the inherent common sense and reasoning ability of the MLLMs are preserved while equipping them with manipulation abilities. The key insight lies in the introduced fine-tuning paradigm, which incorporates object category understanding, affordance prior reasoning, and object-centric pose prediction to stimulate the reasoning ability of MLLMs in manipulation. During inference, an RGB image and text prompt are utilized to predict the end effector’s pose in a chain of thoughts. Additionally, an active impedance adaptation policy is introduced to plan upcoming waypoints in a closed-loop manner after the initial contact is established. To enable better adaptation to real-world scenarios, a test-time adaptation (TTA) strategy for manipulation is designed. Experimental results in both simulation and real-world environments demonstrate the promising performance of ManipLLM. For more details and demonstrations, please visit the article.
Abstract:Robot manipulation relies on accurately predicting contact points and end-effector directions to ensure successful operation. However, learning-based robot manipulation, trained on a limited category within a simulator, often struggles to achieve generalizability, especially when confronted with extensive categories. Therefore, we introduce an innovative approach for robot manipulation that leverages the robust reasoning capabilities of Multimodal Large Language Models (MLLMs) to enhance the stability and generalization of manipulation. By fine-tuning the injected adapters, we preserve the inherent common sense and reasoning ability of the MLLMs while equipping them with the ability for manipulation. The fundamental insight lies in the introduced fine-tuning paradigm, encompassing object category understanding, affordance prior reasoning, and object-centric pose prediction to stimulate the reasoning ability of MLLM in manipulation. During inference, our approach utilizes an RGB image and text prompt to predict the end effector’s pose in chain of thoughts. After the initial contact is established, an active impedance adaptation policy is introduced to plan the upcoming waypoints in a closed-loop manner. Moreover, in real world, we design a test-time adaptation (TTA) strategy for manipulation to enable the model better adapt to the current real-world scene configuration. Experiments in simulator and real-world show the promising performance of ManipLLM. More details and demonstrations can be found at this https URL.
Read the original article
by jsendak | Dec 30, 2023 | Computer Science
In this article, we explore the challenges of 3D modeling from satellite imagery and introduce SUNDIAL, a comprehensive approach to address these challenges using neural radiance fields. Traditional 3D modeling techniques face difficulties in the remote sensing context due to limited multi-view baselines, varying illumination conditions, and scene changes. With SUNDIAL, we aim to jointly learn satellite scene geometry, illumination components, and sun direction using a single-model approach. Our technique incorporates lighting cues and geometric priors from remote sensing literature, enabling us to model physical properties like shadows, scattered sky illumination, and complex illumination of vegetation and water. We evaluate the performance of SUNDIAL against existing NeRF-based techniques and showcase improved scene and lighting disentanglement, novel view and lighting rendering, and accurate geometry and sun direction estimation on challenging satellite scenes. SUNDIAL has the potential to revolutionize 3D reconstruction in areas like environmental science, urban planning, agriculture, and disaster response.
Abstract:3D modeling from satellite imagery is essential in areas of environmental science, urban planning, agriculture, and disaster response. However, traditional 3D modeling techniques face unique challenges in the remote sensing context, including limited multi-view baselines over extensive regions, varying direct, ambient, and complex illumination conditions, and time-varying scene changes across captures. In this work, we introduce SUNDIAL, a comprehensive approach to 3D reconstruction of satellite imagery using neural radiance fields. We jointly learn satellite scene geometry, illumination components, and sun direction in this single-model approach, and propose a secondary shadow ray casting technique to 1) improve scene geometry using oblique sun angles to render shadows, 2) enable physically-based disentanglement of scene albedo and illumination, and 3) determine the components of illumination from direct, ambient (sky), and complex sources. To achieve this, we incorporate lighting cues and geometric priors from remote sensing literature in a neural rendering approach, modeling physical properties of satellite scenes such as shadows, scattered sky illumination, and complex illumination and shading of vegetation and water. We evaluate the performance of SUNDIAL against existing NeRF-based techniques for satellite scene modeling and demonstrate improved scene and lighting disentanglement, novel view and lighting rendering, and geometry and sun direction estimation on challenging scenes with small baselines, sparse inputs, and variable illumination.
Read the original article
